Designing Quality JSON Schema Errors

json_schema_failures.md

Intro

The JSON Schema spec doesn't speak on error construction at all (which is absolutely the right decision).

This means that each library author has to figure out how to construct informative errors individually.

This document is meant to sketch out some best practices.

Example

Schema

{
  "properties": {
    "foo": {
      "items": {
        "type": "string"
      }
    }
  }
}

Data

{
  "foo": [ null ]
}

Some Options

(Examples in Haskell)

1. The simplest possible

exampleError :: Bool
exampleError = False

This makes for totally unhelpful error messages.

2. Only reporting the bare minimum needed to derive anything else

Note: I think it's important that the errors returned by individual validators don't know about the schema-level errors they will eventually be used to produce. That way they can be used in other schemas in the future without causing problems. In this case we achieve that by parameterizing PropertiesValFailure and ItemsValFailure with the schema level error type (which in this case ends up being ValidationFailure)

-- | Declare our the error type for the "properties" validator.
--
-- It's a hashmap of properties keys to the errors that resulted from them.
--
-- It takes a type argument (`err`) because we want to be agnostic about
-- what errors the schema it's eventually used in can produce. That way
-- it can be used by later JSON Schema specifications as well as the current one.
data PropertiesValFailure err = PropertiesValFailure (HashMap Text err)

-- | Declare the error type for the "items" validator.
--
-- It's a hashmap of indexes to the errors that resulted from the data
-- at that index.
data ItemsValFailure err = ItemsValFailure (HashMap Int err)

-- | Declare the error type for the whole schema.
data ValidationFailure
    = InvalidProperties (PropertiesValFailure Failure)
    | InvalidItems (ItemsValFailure Failure)
    -- ^ In a real JSON Schema lib we'd have a lot more errors than this
    -- to handle (allOf, anyOf, etc.) but this is just an example.
    | LeafFailure
    -- ^ LeafFailure is what we use for "type", as well as any other validators
    -- that themselves can't contain errors (like "maximum", "mininum", etc.)
    -- The value of validators like "type" can be derived from the starting schema
    -- and the rest of the error message (e.g.
    -- `InvalidProperties (PropertiesValFailure (HashMap.singleton "quux" LeafFailure))`
    -- would mean that the validator that cause the error is value of the
    -- schema object at the "properties/quux" key). 

-- | Validation produces a list of `ValidationFailure`s, up to one for each top-level
-- validator.
exampleError :: [ValidationFailure]
exampleError =
    [ InvalidProperties (PropertiesValFailure (
          HashMap.singleton "foo" (
              InvalidItems (ItemsValFailure (HashMap.singleton 0 LeafFailure)
              )
          )
      )
    ]

This provides all the information we need to reconstruct what happended, which is good! But the messages are still hard to read at a glance. What other info should probably be included?

Another issue, is "invalid", "failure", or "error" a better name for the errors validators produce? What about the whole schema?

Docs on how the Python jsonschema library handles errors are here: http://python-jsonschema.readthedocs.io/en/latest/errors/