Draft of Hydra API testing tool

readme.md

Hydra API testing

This gist demonstrates a DSL which could be used to describe a Hydra API (could be generalised to fit any hypermedia API as long as the lingo is made not Hydra-specific)

Step 1 (optional)

The core terminology of an API is hidden in the API Documentation. I propose a tool which takes the RDF contents of said documentation and produces a xtext DSL definition which includes all the API-specific bits: classes, properties, operations.

api-dsl-gen \
    --api http://flick-mis-traem.li/apidoc.jsonld \
    --header Authorization: Bearer jwt-token \
    --out flick-mis-traemli-authenticated.api

Note the JWT. I expect one could also produce multiple such DSL flavor if the API Documentation was dynamic in nature. Each could be used for a subset of tests which depend on the caller's authorizations.

This is step could be optional, at which point any API terms would not be available for IDE suggestions but the general syntax proposed should still allow to exercise an API using links and forms provided by hand.

Step 2

Check the other files for detaild examples of how the test "scenarios" could be structured

Notes on processing rules

I recognize that there are two possible ways to construct the scenarios:

1. Nesting subsequent steps
2. Hoisting steps to top level

I think that both approaches can be supported and complement each other.

When instructions to follow links or invoke operations are nested, they should be attempted unconditionally. This way unsafe operation, such as DELETE can be exercised within a well-defined execution path.

Otherwise, possible interactions are selected from those defined at top level. For example, the first time a resource of type fmt:ProblemReport is encountered, its operations and links are looked up at top level and executed. It's important that each such top-level path is only ever ran once.

coverage.md

Coverage

There could be two separate coverage metrics.

Representation metrics

To verify the contents of resources

I'm not so sure about, but any verification like Expect property xyz (value) could increase this metric.

Interaction metrics

To try executing operation and following links.

TBD: Maybe links and operations should be split?

With regard to operations, the total possible points should be calculated based on:

1. Invoking the operation with various payloads, determined by conting various possibilities of valid and invalid request bodies
2. Following links

flick-mis-traemli-authenticated.api

Supported Class https://flick-mi-traem.li/vocab#Entrypoint {
    Property http://schema.org/name

    Link https://flick-mi-traem.li/vocab#openProblems

    Operation https://flick-mi-traem.li/vocab#CreateVehicleProblemReport
}

less-nesting.md

TBD: during execution some links may be absent but appear later. how to discover those scenarios?

TBD: what about unsafe operations

API ./flick-mis-traemli-authenticated.api

PREFIX fmt: <https://flick-mi-traem.li/vocab#>

Resource fmt:Installation {
    Expect property fmt:installation {
        // TODO: cannot distinguish operations by ID ATM
        Invoke operation fmt:Install {
            Expect status 204
            Expect header Location [uri]

            // TODO: possibly response with Location header should be implicitly fetched
            Fetch [uri]
        }
    }
}

Resource fmt:Entrypoint {
    Follow link fmt:openProblems {
        Expect resource collection of fmt:ProblemReport
    }

    Expect property fmt:newProblem {
        Expect operation fmt:CreateVehicleProblemReport
        Expect operation fmt:CreateLocationProblemReport
        Expect no operation fmt:SomethingInaccessibleToCurrentUser
    }

    Follow link fmt:dangerZone
}

Resource collection of fmt:ProblemReport {
    // TODO
}

Resource fmt:NukeDatabase {
    Expect property fmt:entrypoint
}

Operation fmt:DropDatabase {
    Invocation {
        Expect status 200
    }
}

Operation fmt:CreateVehicleProblemReport {
    Invocation {
        Body application/turtle
        ```
        @prefix fmt: <http://ld.bernmobil.ch/vocab/ticket#> .
        @prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
        @prefix schema: <http://schema.org/> .

        [] a <http://flick-mis-traem.li/vocab#VehicleProblemReport> ;
            rdfs:label "broken window" ;
            fmt:damageType "http://ld.bernmobil.ch/code/kat3" ;
            schema:location "http://ld.bernmobil.ch/ticket/fahrzeug/20279" ;
            rdfs:comment "" .
        ```

        Expect status 201
        Expect header Location [newProblem]

        // no fetch. ignore further
    }
}

Operation fmt:CreateLocationProblemReport {
    Invoke {
        Body application/ld+json
        ```
        {
          "@type":"http://flick-mis-traem.li/vocab#LocationProblemReport",
          "http://www.w3.org/2000/01/rdf-schema#label":"test",
          "http://www.w3.org/2000/01/rdf-schema#comment":"",
          "http://schema.org/location":"https://ld.geo.admin.ch/transportation/stop/8581033",
          "http://ld.bernmobil.ch/vocab/ticket#damageType":"http://ld.bernmobil.ch/code/kat2"
        }
        ```

        Expect status 201
        Expect header Location [newProblem]

        Fetch [newProblem] {
            Expect property schema:label "test"
            Expect property schema:location <https://ld.geo.admin.ch/transportation/stop/8581033>
            Expect property http://ld.bernmobil.ch/vocab/ticket#damageType <http://ld.bernmobil.ch/code/kat2>

            Invoke operation fmt:DeleteProblem {
                Expect status 200
            }
        }
    }

    Invocation {
        Body application/ld+json
        ```
        {
          "@type":"http://flick-mis-traem.li/vocab#LocationProblemReport",
          "http://www.w3.org/2000/01/rdf-schema#comment":"",
          "http://schema.org/location":"https://ld.geo.admin.ch/transportation/stop/8581033",
          "http://ld.bernmobil.ch/vocab/ticket#damageType":"http://ld.bernmobil.ch/code/kat2"
        }
        ```

        Expect status 400
        Expect no header Location
    }
}

nested-operations.md

TODO: during execution some links may be absent but appear later. how to distinguish those scenarios?

TODO: what about unsafe operations

API ./flick-mis-traemli-authenticated.api

PREFIX fmt: <https://flick-mi-traem.li/vocab#>

Resource fmt:Installation {
    // TODO: cannot distinguish operations by ID ATM
    Invoke operation fmt:Install {
        Expect status 204
        Expect header Location [uri]

        // TODO: possibly response with Location header should be implicitly fetched
        Fetch [uri]
    }
}

Resource fmt:Entrypoint {
    Follow link fmt:openProblems {
        Expect resource collection of fmt:ProblemReport
    }

    Property fmt:newProblem {
        // TODO: possibly move operations to top level
        Invoke operation fmt:CreateLocationProblemReport (
            ```application/ld+json
            {
              "@type":"http://flick-mis-traem.li/vocab#LocationProblemReport",
              "http://www.w3.org/2000/01/rdf-schema#label":"test",
              "http://www.w3.org/2000/01/rdf-schema#comment":"",
              "http://schema.org/location":"https://ld.geo.admin.ch/transportation/stop/8581033",
              "http://ld.bernmobil.ch/vocab/ticket#damageType":"http://ld.bernmobil.ch/code/kat2"
            }
            ```
        ) {
            Expect status 201
            Expect header Location [newProblem]

            // TODO: without `Fetch` would verify on the returned representation, if any
            Fetch [newProblem] {
                Expect property schema:label "test"
                Expect property schema:location <https://ld.geo.admin.ch/transportation/stop/8581033>
                Expect property http://ld.bernmobil.ch/vocab/ticket#damageType <http://ld.bernmobil.ch/code/kat2>
            }
        }

        // TODO can we narrow down suggestions by context
        Invoke operation fmt:CreateVehicleProblemReport (
            ```application/turtle
            @prefix fmt: <http://ld.bernmobil.ch/vocab/ticket#> .
            @prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
            @prefix schema: <http://schema.org/> .

            [] a <http://flick-mis-traem.li/vocab#VehicleProblemReport> ;
                rdfs:label "broken window" ;
                fmt:damageType "http://ld.bernmobil.ch/code/kat3" ;
                schema:location "http://ld.bernmobil.ch/ticket/fahrzeug/20279" ;
                rdfs:comment "" .
            ```
        ) {
            Expect status 201
            Expect header Location [newProblem]

            // no fetch. ignore further
        }
    }

    Follow link fmt:dangerZone
}

Resource collection of fmt:ProblemReport {
    // TODO
}


Resource fmt:NukeDatabase {
    Property fmt:entrypoint {
        Invoke operation fmt:DropDatabase {
            Expect status 200
        }
    }
}