Validate Your PHP API Tests Against OpenAPI Definitions

Am I proud of this montage? You bet I am (by the author)

OpenAPI is a specification intended to describe RESTful APIs in JSON and YAML, with the aim of being understandable by humans and machines alike.

OpenAPI definitions are language-agnostic and can be used in a lot of different ways:

An OpenAPI definition can be used by documentation generation tools to display the API, code generation tools to generate servers and clients in various programming languages, testing tools, and many other use cases.

– The OpenAPI Specification

In this article, we will see how to combine OpenAPI 3.0.x definitions with integration tests to validate whether an API behaves the way it's supposed to, using the OpenAPI HttpFoundation Testing package.

We will do so in a fresh Laravel installation, for which we'll also generate a Swagger UI documentation using the L5 Swagger package.

I will first elaborate a bit further on why this is useful, but if you're just here for the code, you're welcome to skip ahead and go to the A Laravel example section straight away.

The issue

APIs are pretty common nowadays, and when we're lucky they come with some form of documentation that helps us find our way around the endpoints. These documentations come in many shapes and flavours (some tastier than others), but one thing they've got in common in that they need to be updated every time the API they describe changes.

To many developers, maintaining an API's documentation feels like extra homework when they've already passed the exam; it's boring, sometimes tedious, and often unrewarding. Some strategies can help, like using annotations to keep the code and the documentation in one place; but those are often annoying to write still, and even the most willing developer is not immune to an oversight that won't necessarily be caught by coworkers.

The usual outcome is that, one way or another, the documentation and the API become out of sync, leading to confused consumers.

Another aspect of API maintenance is ensuring that no endpoint stops functioning the way it's supposed to; regressions will be introduced eventually, and without a proper testing strategy they might go unnoticed for a while.

A way to avoid this is to implement integration tests that will automatically check that the API's behaviour is correct, and that recently introduced changes have not had unintended consequences. This is fine, but still doesn't provide any guarantee that the expectations set in the integration tests are exactly the same as the ones inferred from the documentation.

If only there was a way to ensure that they perfectly reflect each other…

A solution

We are now assuming that we've got an API documentation and some integration tests, and we'd like to align their expectations somehow.

The OpenAPI specification has become a popular choice to describe APIs over time, but whether we use it or not doesn't change the fact that the corresponding definitions need to be maintained; in other words, using OpenAPI does not automagically make the aforementioned issues go away.

What sets OpenAPI apart, however, is that it's used as the base layer for a growing number of tools that make the specification useful far beyond the mere documenting side of things.

One of these tools built for the PHP ecosystem and maintained by The PHP League is OpenAPI PSR-7 Message Validator, a package for validating HTTP requests and responses implementing the PSR-7 standard against OpenAPI definitions.

The idea is essentially to take HTTP requests and responses, and make sure they match one of the operations described in an OpenAPI definition.

Can you see where this is going?

We could basically use this package to add an extra layer on top of our integration tests, that will take the API responses obtained in the tests and make sure they match the OpenAPI definitions describing our API.

If they don't, the tests fail.

This is what it looks like as a fancy diagram:

By the author

The OpenAPI definition describes the API, and the tests use the OpenAPI definition to make sure the API actually behaves the way the definition says it does.

All of a sudden, our OpenAPI definition becomes a reference for both our code and our tests, thus acting as the API's single source of truth.

PSR-7

You might have noticed a small detail in the previous section: the OpenAPI PSR-7 Message Validator package only works for – it's in the name – PSR-7 messages. The issue here is that not all frameworks support this standard out of the box; as a matter of fact, a lot of them use Symfony's HttpFoundation component under the hood, whose requests and responses do not implement that standard by default.

The Symfony folks have got us covered though, as they've developed a bridge that converts HttpFoundation objects to PSR-7 ones, as long as it's given a PSR-7 and PSR-17 factory to do so, for which they suggest to use Tobias Nyholm's PSR-7 implementation.

All of these pieces form a jigsaw puzzle that the OpenAPI HttpFoundation Testing package offers to assemble for us, allowing developers to back their integration tests with OpenAPI definitions in projects leveraging the HttpFoundation component.

Let's see how to use it in a Laravel project, which falls into this category.

A Laravel example

The code contained in this section is also available as a GitHub repository.

First, let's create a new Laravel 8 project, using Composer:

$ composer create-project --prefer-dist laravel/laravel openapi-example "8.*"

Enter the project's root folder and install a couple of dependencies:

$ cd openapi-example
$ composer require --dev osteel/openapi-httpfoundation-testing
$ composer require darkaonline/l5-swagger

The first one is the OpenAPI HttpFoundation Testing package mentioned earlier, that we install as a development dependency as it's intended to be used as part of our test suite.

The second one is L5 Swagger, a popular package bringing Swagger PHP and Swagger UI to Laravel. We actually don't need Swagger PHP here, as it uses Doctrine annotations to generate OpenAPI definitions and we're going to write our own manually instead. We do need Swagger UI, however, and the package conveniently adapts it to work with Laravel.

To make sure Swagger PHP doesn't overwrite the OpenAPI definition, let's set the following environment variable in the

.env

file at the root of the project:

L5_SWAGGER_GENERATE_ALWAYS=false

Create a file named

api-docs.yaml

in the

storage/api-docs

folder (which you need to create), and add the following content to it:

This is a simple OpenAPI definition describing a single operation – a

GET

request on the

/api/test

endpoint, that should return a JSON object containing a required

foo

key.

Let's check whether Swagger UI displays our OpenAPI definition correctly. Start PHP's development server with this

artisan

command, to be run from the project's root:

$ php artisan serve

Open localhost:8000/api/documentation in your browser and replace

api-docs.json

with

api-docs.yaml

in the navigation bar at the top (this is so Swagger UI loads up the YAML definition instead of the JSON one, as we haven't provided the latter).

Hit the enter key or click Explore – our OpenAPI definition should now be rendered as a Swagger UI documentation:

Expand the

/test

endpoint and try it out – it should fail with a

404 Not Found

error, because we haven't implemented it yet.

Let's fix that now. Open the

routes/api.php

file and replace the example route with this one:

Route::get('/test', function (Request $request) {
    return response()->json(['foo' => 'bar']);
});

Go back to the Swagger UI tab and try the endpoint again – it should now return a successful response.

Time to write a test! Open

tests/Feature/ExampleTest.php

and replace its content with this one:

Let's unpack this a bit. For those unfamiliar with Laravel,

$this->get()

is a test method provided by the

MakesHttpRequests

trait that essentially performs a

GET

request on the provided endpoint, executing the request's lifecycle without leaving the application. It returns a response that is identical to the one we would obtain if we'd perform the same request from the outside.

We then create a validator using the

Osteel\OpenApi\Testing\ResponseValidatorBuilder

class, to which we feed the YAML definition we wrote earlier via the

fromYaml

static method (the

storage_path

function is a helper returning the path to the

storage

folder, where we stored the definition).

Had we had a JSON definition instead, we could have used the

fromJson

method; also, both methods accept YAML and JSON strings respectively, as well as files.

The builder returns an instance of

Osteel\OpenApi\Testing\ResponseValidator

, on which we call the

get

method, passing the path and the response as parameters (

$response

is a

Illuminate\Testing\TestResponse

object here, which is a wrapper for the underlying HttpFoundation object, which can be retrieved through the

baseResponse

public property).

The above is basically the equivalent of saying I want to validate that this response conforms to the OpenAPI definition of a

GET

request on the

/test

path.

It could also be written this way:

$result = $validator->get('/test', $response->baseResponse);

That's because the validator has a shortcut method for each of the HTTP methods supported by OpenAPI (

GET

POST

PUT

PATCH

DELETE

HEAD

OPTIONS

and

TRACE

), to make it simpler to test responses for the corresponding operations.

Note that the specified path must exactly match one of the OpenAPI definition's paths.

You can now run the test, which should be successful:

$ ./vendor/bin/phpunit tests/Feature

Open

routes/api.php

again, and change the route for this one:

Route::get('/test', function (Request $request) {
    return response()->json(['baz' => 'bar']);
});

Run the test again; it should now fail, because the response contains

baz

instead of

foo

, and the OpenAPI definition says the latter is expected.

Our test is officially backed by OpenAPI!

The above is obviously an oversimplified example for the sake of the demonstration, but in a real situation a good practice would be to overwrite the

MakesHttpRequests

trait's

call

method, so it performs both the test request and the OpenAPI validation.

As a result, our test would now be a single line:

$this->get('/api/test');

This could be implemented as a new

MakesOpenApiRequests

trait that would "extend" the

MakesHttpRequests

one, and that would first call the parent

call

method to get the response. It would then work out the path from the URI, and validate the response against the OpenAPI definition before returning it, for the calling test to perform any further assertions as needed.

Conclusion

While the above setup is a great step up in improving an API's robustness, it is no silver bullet; it requires that every single endpoint is covered with integration tests, which is not easily enforceable in an automated way, and ultimately still requires some discipline and vigilance from the developers. It may even feel a bit coercive to some at first, since as a result they are basically forced to maintain the documentation in order to write successful tests.

The added value, however, is that said documentation is now guaranteed to be much more accurate, leading to happy consumers who will enjoy an API which is less likely to act erratically; this, in turn, should lead to less frustrated developers, who shall spend less time hunting down pesky discrepancies.

All in all, making OpenAPI definitions the single source of truth for both the API documentation and the integration tests is in itself a strong incentive to keep them up to date; they naturally become a priority, where they used to be an afterthought.

As for maintaining the OpenAPI definition itself, doing so manually can admittedly feel a bit daunting. Annotations are a solution, but I personally don't like them and prefer to maintain a YAML file directly. IDE extensions like this VSCode one make it much easier, but if you can't bear the sight of a YAML or JSON file, you can also use tools like Stoplight Studio to do it through a more user-friendly interface.

And since we're talking about Stoplight*, this article about API Design-First vs Code First by Phil Sturgeon is a good starting point for API documentation in general, and might help you choose an approach to documenting that suits you.

* I am not affiliated with Stoplight in any way

Resources