Comparing openapi_first and committee

openapi_first-committee-comparison.md

This is a feature comparison of openapi_first and committee

I will try to keep this updated. Please create an issue if you see something wrong/missing here.

JSON Schema validation

Most of these differences are because openapi_first uses json_schemer, which has build in support for OpenAPI 3.1 where committee/openapi_parser currently ships it's own implementation.

Feature	openapi_first	committee
Support OpenAPI 3.1 (Published 15 February 2021)	✅	❌ issue
Validate enums	✅	❌ issue
Support explode: true in array query parameters	✅	❌
Support if/else in JSON Schemas	✅	❌
Support writeOnly/readOnly	✅ (docs)	❌ issue
Support all format validations (like format: date)	✅	❌ issue
Support nullable: true (removed in OpenAPI 3.1)	✅	✅
Support type arrays (type: [string, 'null']) (OpenAPI 3.1)	✅	❌
Support discriminator	✅	✅

Parsing parameters

Feature	openapi_first	committee
Parse and validate query parameters that include brackets (?names[]=1")	✅	❌
Parse parameters exactly as described. Support all explode and style options.	✅	❌ issue issue
Convert simple parameter values (integer, number)	✅	✅
Support named nested parameters (name: 'filter[thing]')	✅	✅ issue

Other

Feature	openapi_first	committee
$ref just works	✅	❌ issue issue issue
Validate request body	✅	✅
Validate request parameters	✅	✅
Validate response body	✅	✅
Validate response headers	✅	⚠️ committee does not complain about missing required response headers issue
Coverage	❌	✅
Test assertions	✅	✅
Option to raise exception if request/response is invalid	✅	✅
Option to raise exception if request is not specified	✅	✅
Complain about unkown response body content type	✅	⚠️ (committee ignores unknown response bodies by default unless you use the strict option)
Convert date, date-time request values to Date, DateTime	❌	✅
Support custom error response	✅	✅
Add parsed parameters, body to rack env	✅	✅
Add OpenAPI request object to rack env to enable runtime inspection	✅	❌
Middleware has configurable callback when validation error occurs	☑️ (use hooks)	✅

Error messages during development

Error: Referenced file could not be found

committee:

NoMethodError:
    undefined method `set_path_item_to_operation' for an instance of OpenAPIParser::Schemas::OpenAPI
  # ./spec/support/web_helpers.rb:6:in `app'

or

  OpenAPIParser::MissingReferenceError:
    './paths/access-control_roles.yaml#/components/paths/access-control_roles' was referenced but could not be found
  # ./spec/support/web_helpers.rb:6:in `app'

openapi_first:

  OpenapiFirst::FileNotFoundError:
    Problem while loading file referenced in openapi/openapi.yaml: File not found /Users/ahaller/src/access-control-api/openapi/schemas/PermissionObject.yaml
  # /Users/ahaller/src/openapi_first/lib/openapi_first/json_refs.rb:144:in `load_referenced_file'
  # /Users/ahaller/src/openapi_first/lib/openapi_first/json_refs.rb:137:in `dereference_file'
  # /Users/ahaller/src/openapi_first/lib/openapi_first/json_refs.rb:122:in `referenced_value'
...

JSON Schema validation error: unknown properties when using additionalProperties: false

committee:

Committee::InvalidResponse:
  #/components/schemas/RoleInIndex does not define properties: type, attributes, relationships

openapi_first.

OpenapiFirst::ResponseInvalidError:
  Response body is invalid: object property at `/data/0/type` is not defined and schema does not allow additional properties. object property at `/data/0/attributes` is not defined and schema does not allow additional properties. object property at `/data/0/relationships` is not defined and schema does not allow additional properties. object at `/data/0` is missing required properties: name, nameShort, description, iwfmUserGroupUrl, createdAt, updatedAt, isDeletable, isReadonly, isAdmin, forceOnAllUsers, category. ... (16 errors total)

Have you compared performance issues (latancy and troughput)?

Hi @andrew-platonov there are some benchmarks here

I just added a benchmark using wrk. Here is the output comparing committee and openapi_first. Note that both are using response validation, which I usually would not enable in production:

Results