Request Validation with Akka-HTTP and Cats

akka_http_and_cats_validation.md

Request Validation with Akka-HTTP and Cats

Greg Beech wrote an article on Akka HTTP Entity Validation. This post is on the same topic, except we are going to use typeclass instances, as opposed to subtyping, for form validations.

Sample Problem: User Regsitration

To start with, let us suppose that we are working on some sort of user management service using Scala/Akka. However, here we are only going to focus on the registration part, in which the client is expected to send a user data, in JSON format; and the server needs to validate them, and save the user if the validations succeed, or reject the request with all the errors detected. That said, we should keep in mind that our validations may be used through out the entire codebase, not just in the user registration endpoint.

The rules for our validation are as follows:

1. All fields are required.
2. Each field must have the correct type.
3. All strings must be non-empty.
4. Classic password validations (which we are not going to emphasize much).
5. A user should have at least one hobby (see #1).
6. Only people of ages 18 or above are allowed to register (it's not for kids).

Note that in a real-world project, registration validations are usually more complicated than this list, but for our current purposes, this should do.

Data Model and Route Implementation

Our user model has the usual attributes present in most registration forms:

final case class User(
  username: String,
  password: String,
  firstName: String, 
  lastName: String,
  age: Int, 
  hobbies: Vector[String])

In order to parse the JSON data and convert it into a User instance and vice-versa, we need to tell Akka-HTTP how to (un)marshall between JSON and User:

object RegistrationJsonSupport extends SprayJsonSupport with DefaultJsonProtocol {
  implicit val userFormat = jsonFormat6(User)
  // ... more formats here ...
}

Then we bring that into the scope and use it in our route:

import RegistrationJsonSupport._

val route = post {
  entity(as[User]) { user =>
    // For simplicity, let us agree that this operation never fails.
    saveUser(user)
    
    complete(s"User ${user.username} has been successfully registered.")
  }
}

This has added advantages: Akka-HTTP ensures that non-optional fields are provided by the client, and each field will have the correct type as specified in the model. Any requests that do not comply with these structural rules will be rejected. This means our current code already got the first 2 validation rules above covered.

Fields Validation

The next thing we need to do is implement the different validations for the fields.

If you are interested only in the form validations and how to integrate them with Akka-HTTP, you can skip this section.

The remaining validation rules are all about checking for non-emptiness and minimum values, and password validations. We can write our own utilities to generalize some of these things:

trait FieldValidator {
  trait Required[F] extends (F => Boolean)
  
  trait Minimum[F] extends ((F, Int) => Boolean)
  
  def required[F](field: F)(implicit req: Required[F]): Boolean =
    req(field)

  def minimum[F](field: F, limit: Int)(implicit min: Minimum[F]): Boolean =
    min(field, limit)

  implicit val minimumStringLength: Minimum[String] = _.length >= _

  implicit val minimumInteger: Minimum[Int] = _ >= _

  implicit val requiredString: Required[String] = _.nonEmpty

  // Warning: I haven't tried compiling this part.
  implicit val requireVector: Required[Vector[String]] = _.forall(required[String])
}

If that looks like a bunch of typeclasses and their instances, that's because it is. I've got two main motivations for suggesting this approach:

1. Type-safety. Having minimum and required take Any values on which to pattern match to determine the specific type of input is not type-safe.
2. Ad-hoc Polymorphism. You do not have to invoke different functions for the same type of validations (minimumInt, minimumString, etc.). If you want to check for, say, a minimum, you can just call the minimum function regardless of the type of field, providing, of course, that the typeclass has an (implicit) instance for that field in scope. Sometimes you might need to specify the type of the field to check (e.g. mimimum[Int], required[String]).

We can then use these utilities in our field validation functions:

import cats.data.ValidatedNec
import cats.implicits._

trait FieldValidator {
  // previous code goes here...
  
  def validateRequired[F: Required](field: F, fieldName: String): ValidationResult[F] =
    Either.cond(
      required(field),
      field,
      EmptyField(fieldName)).toValidatedNec

  def validateMinimum[F: Minimum](field: F, fieldName: String, limit: Int): ValidationResult[F] =
    Either.cond(
      minimum(field, limit),
      field,
      BelowMinimum(fieldName, limit)).toValidatedNec
      
  def validatePassword(password: String): ValidationResult[String] = ...    
      
  // more validations here
}

In case you didn't know what [F: Required] indicates, it's a Scala feature known as context bounds. Put simply, the signatures desugar to:

def validateRequired[F](field: F, fieldName: String)(implicit required: Required[F]): ValidationResult[F] = ???
def validateMinimum[F](field: F, fieldName: String, limit: Int)(implicit min: Minimum[F]): ValidationResult[F] = ???

We use Either.cond to handle the results of the validations. EmptyField and BelowMinimum are case classes representing errors:

sealed trait RequestValidation {
  def errorMessage: String
}

object RequestValidation {
  final case class EmptyField(fieldName: String) extends RequestValidation {
    override def errorMessage = s"$fieldName is empty"
  }

  final case class BelowMinimum(fieldName: String, limit: Int) extends RequestValidation {
    override def errorMessage = s"$fieldName is below the minimum of $limit"
  }
  
  // ... more error messages here...
}

Let's go back to our field validation functions. You can see that the resulting Eithers from the Either.conds call toValidatedNec. This method turns an Either into a ValidatedNec from Cats. ValidatedNec is just like Either, but far more useful when accumulating multiple errors. However, if you look at the signature of the functions, the results are actually VaidationResults, a type that we'll talk about in the next section.

Form Validation

Our form validator will try to validate all the fields in a given request. It can produce two possible outcomes: the validated object indicating that the validation went successfully, or an accumulation of errors. For this, we've got two options: use Either or Cats's ValidatedNec. Cats documentation for Validated recommends to use Validated (and consequently, ValidatedNec) for the accumulation of errors. In this post, we are going for ValidatedNec.

Just like Either, ValidatedNec has left and right projections, with left used to represent errors. We already know the type of errors we want, and we don't want to keep on specifying it, so we'll make a type alias that will fill in ValidatedNec's left projection with RequestValidation.

And that's where the ValidationResult we saw in the previous section comes in:

type ValidationResult[A] = ValidatedNec[RequestValidation, A]

Let's also add a FormValidation type alias for a function that lifts a form (of any type) to ValidationResult (which would allow us to do point-free style programming later on):

type FormValidation[F] = F => ValidationResult[F]

Now that we have the types in place, we can write a generic form validator:

def validateForm[F, A](form: F)(f: ValidationResult[F] => A)
    (implicit formValidation: FormValidation[F]): A =
  f(formValidation(form))

I forgot to tell you that FormValidation can be treated as a typeclass. And that means we can apply validateForm to any form of type F as long as there is a FormValidation instance for that form in scope.

Here's our form validator, with the typeclass instances:

trait FormValidator {
  type ValidationResult[A] = ValidatedNec[RequestValidation, A]
  type FormValidation[F] = F => ValidationResult[F]

  def validateForm[F, A](form: F)(f: ValidationResult[F] => A)
      (implicit formValidation: FormValidation[F]): A =
    f(formValidation(form))

  implicit lazy val registrationFormValidation: FormValidation[User] = {
    case User(username, password, firstName, lastName, address, age) => (
      validateRequired(username, "Username"),
      validatePassword(password, "Password"),
      validateRequired(firstName, "First Name",
      validateRequired(lastName, "Last Name"),
      validateRequired(age, "Age"),
      validateRequired(hobbies, "Hobbies")
      validateMinimum(age, "Age", 18))).mapN(User)
  }
  
  // ...other implicit typeclass instances here...
  implicit lazy val userUpdateFormValidation: FormValidation[User] = ???
  implicit lazy val userDeletionFormValidation: FormValdiation[String] = ???
}

mapN would take our chain of validations and accumulate all the errors found in the process, wrapping them all in the Invalid projection (ValidatedNec's equivalent to Either's Left). If no errors are found, it will return a Valid result (ValidatedNec's equivalent to Either's Right) with a user instance as the content.

We can then incorporate it into our route:

val route = post {
  entity(as[User]) { user =>
    validateForm(user) { 
      case Valid(_) =>
        saveUser(user)
        complete(s"User ${user.username} has been successfully registered.")
      case Invalid(failures) => complete(internalError(
          failures.iterator.map(_.errorMessage).mkString("\n")))
    } 
  }
}

The only remaining thing to do is the removal of that extra call to validateForm, which adds another layer of indentation, and explicit handling of the validation results.

To do that we may have to implement our custom unmashaller that does the validation. The problem is, Greg's custom unmarshaller, implemented using implicit conversions, wouldn't work in our case because our validations are not part of the entity. Also we will need to declare implicit parameters for the instances of our FormValidation typeclasses.

Fortunately, there is another way to circumvent the issue, and that is to add extension methods to the default unmarshaller:

implicit class ValidationRequestMarshaller[A](um: FromRequestUnmarshaller[A]) {
  def validate(implicit validation: FormValidation[A]) = um.flatMap { _ => _ => entity =>
    validateForm(entity) {
      case Valid(_) => Future.successful(entity)
      case Invalid(failures) =>
        Future.failed(new IllegalArgumentException(
          failures.iterator.map(_.errorMessage).mkString("\n")))
    }
  }
}

We are using implicit class instead of direct implicit conversion (implicit def). This implicit class effectively adds a validate method to a FromRequestUnmarshaller. Note that validate also contains the implicit parameter I mentioned above.

Here's the final, much better version of our route:

import validators._   // perhaps a package object that extends both FieldValidator and FormValidator

...

val route = post {
  entity(as[User].validate) { user =>
    saveUser(user)
    complete(s"User ${user.username} has been successfully registered.")
  }
}

Sometimes making a few small functions and calling them within complete is a useful model to explicitly compose computation in a centralized manner.

E.g.,

def saveUser(user: User): Future[User] = Future.successful(user)
def savedMessage(user: User) =
    s"User ${user.username} has been successfully registered."

def validateUser(user: User): Future[User] = Future.successful(user)

def saveUserRoutes(): Route = {
    pathPrefix("create-user") {
      post {
        entity(as[User]) { user =>
          complete {
            for {
              validatedUser <- validateUser(user)
              user <- saveUser(validatedUser)
            } yield savedMessage(user)
          }
        }
      }
    }
  }

@mpkocher You can do this too, but I did not want to bother too much with whatever is inside the entity block in this case (i.e. mine looked procedural, with saveUser clearly returning a Unit, as opposed to functional), as I just wanted to show that validations could happen before that point. But maybe I missed your point.

Hi @melvic-ybanez That is an interesting way of using Cats' Validated however I found few problems using this approach.
1- You can't have more than one validation by field (you could create a validation that validates N situations, but it isn't so good if you have many validations for one field)
2 - You can't validate the attributes of the references
3 - If you want not to validate some field you need to create a dummy validation, only for returning the field value for the mapN function
4 - Although I thought the final result was fancy, I tend to think that the routing layer isn't the appropriate place to validate your model, having in mind a routing layer should know only about routes. Also if you have other validations on another layer you can't mix both validations having only one feedback with all of them.
5 - If you have the creating and updating methods nearly in the same class, you will need to import each implicit practically above of your validation calling to avoid resolution conflict.

What do you think about this feedback? If you agree with some point, have you found an alternative for that?

@emersonmoura

1. You can have multiple validations for one field. Note that every field validation is also a ValidationResult. That means you can compose multiple field validations the same way you combined them in the FormValidation (i.e form validation is just a function that yields another validation result). In fact, a field that has multiple parts can be treated as a sub-form by itself. For example you can validate a phone number field with the following:

def validateNumber: FormValidation[PhoneNumber] = {
  case PhoneNumber(code, number) => (
    validateRequired(countryCode, "Country Code"),
    validateNumberFormat(number, "Number")
  ).mapN(PhoneNumber)
}

Then you can use that validation for the phone number field in the user form validation

2. Does the above answer this as well?

3. You can choose not to validate some fields:

def validateCat: FormValidation[Cat] = {
  case cat @ Cat(name, age, remainingLives) => (
    validateRequired(name, "Name"),
    validateMax(remainingLives, 9, "Remaining Lives")
  ) .mapN { case (newName, newRemainingLives) => 
    cat.copy(name = newName, remainingLives = newRemainingLives)
  }
}

4. You can move the validation elsewhere. The logic isn't necessarily coupled into the routing. And you can compose multiple validations as shown in my answer to #1.

5. By implicits are you talking about Minimum and Required? If so then you can globalize them by putting them in the companion object of their respective traits:

trait Required[F] extends (F => Boolean)

object Required {
  // no need to import this implicit
  implicit val requiredString: Required[String] = _.nonEmpty
}

You can always decide to override them with you own custom local implicits.

Hi @melvic-ybanez thank you for your fast answer.
Items 2,3 and 4: Perfect! I understood them. but your answer for the first item looks like more a reference validation than multiple validations for the same field. My question about many validations was about the same field, for example:

def validateCat: FormValidation[Cat] = {
  case cat @ Cat(name, age, remainingLives) => (
    validateRequired(name, "Name"),
    validateMax(name, 9, "Name should be lower than 9"),
    validateMin(name, 3, "Name should be greater than 3")
  ) .mapN { case (name1, name2, name3) => 
    cat.copy(name = name1)
  }
}

As you can see, If I needed to have more than one validation for the same field it would not be so fancy (obviously, according to my example)

By implicit I'm talking about this part:

implicit lazy val registrationFormValidation: FormValidation[User] = ...
implicit lazy val userUpdateFormValidation: FormValidation[User] = ...
....

We can't import all of them at the top of our class if we need to validate more then one context (create and update for example).
We may need to import that close to the validateForm method (please, don't consider the code duplication)

val route = post {
  entity(as[User]) { user =>
   import FormValidator.registrationFormValidation
    validateForm(user) { 
      case Valid(_) =>
        saveUser(user)
        complete(s"User ${user.username} has been successfully registered.")
      case Invalid(failures) => complete(internalError(
          failures.iterator.map(_.errorMessage).mkString("\n")))
    } 
  }
} ~  put {
  entity(as[User]) { user =>
   import FormValidator.userUpdateFormValidation
    validateForm(user) { 
      case Valid(_) =>
        updateUser(user)
        complete(s"User ${user.username} has been successfully updated.")
      case Invalid(failures) => complete(internalError(
          failures.iterator.map(_.errorMessage).mkString("\n")))
    } 
  }
}

I solved it passing a validation group class as a type:

implicit lazy val registrationValidation: FormValidation[User, Class[Create]] 

And I can import all implicit methods at the top of my class, having in mind that I can validate doing something like this:

validateForm(user, classOf[Create])

What is your opinion about these topics?

@emersonmoura

1. Yeah, you do it that way, or at least that's one easy way to do it.

2. For the implicits, you can model FormValidator as a typeclass by adding a type parameter and a summoner

trait FormValidator[V] {
  def validateForm[F, A](form: F)(f: ValidationResult[F] => A)
      (implicit formValidation: FormValidation[F])
}

object FormValidator {
  def apply[A](implicit f: FormValidator[A]): FormValidator[A] = f
}

You then provide implicits for the create and update in the companion objects. Since you don't really need to use the instance of the type param V inside the FormValidator, it will be treated as a phantom type. That means it's present but not used, making a difference only at the type-level.

// You can't instantiate these classes. They only serve as phantoms.
final abstract class Create
final abstract class Update

implicit val createFormValidtor: FormValidator[Create] = ...
implicit val updateFormValdiator: FormValidator[Update] = ...

Now you can use them like this:

FormValidator[Create].validateForm(form)

Hi @melvic-ybanez,
So we have the same vision about this part. Cool!

Did you ever evaluate using refined types?

@rcardin Did you mean to ask if I've ever used refined types for validation? Yes, I have used refined types, but not at the time of this article's writing.

I meant, instead of creating your own function to validate date, maybe, it's possible to use refined types directly :)

@rcardin Yes. For compile-time values, refined types are even better because your code won't compile if the values are invalid. For runtime values, you might want to use refineV, which returns an Either and can be integrated with Cats validation.

Hey, I managed to use your approach successfully. However, I've got a question. What if I need to return a JSON error in case of a validation error? Is it possible?

Hey, I managed to use your approach successfully. However, I've got a question. What if I need to return a JSON error in case of a validation error? Is it possible?

Hi @rcardin. Yes, it is. All you have to do is transform the request validation to json. One way to do it is to use json encoder. In fact, the advantage of returning an algebraic data type is the ability to interpret it in different ways in the end.

Maybe I missed the point.

implicit class ValidationRequestMarshaller[A](um: FromRequestUnmarshaller[A]) {
  def validate(implicit validation: FormValidation[A]) = um.flatMap { _ => _ => entity =>
    validateForm(entity) {
      case Valid(_) => Future.successful(entity)
      case Invalid(failures) =>
        Future.failed(new IllegalArgumentException(
          failures.iterator.map(_.errorMessage).mkString("\n")))
    }
  }
}

The validate method can only return a Future, and futures completed exceptionally can only wrap an exception.

Maybe I missed the point.

implicit class ValidationRequestMarshaller[A](um: FromRequestUnmarshaller[A]) {
  def validate(implicit validation: FormValidation[A]) = um.flatMap { _ => _ => entity =>
    validateForm(entity) {
      case Valid(_) => Future.successful(entity)
      case Invalid(failures) =>
        Future.failed(new IllegalArgumentException(
          failures.iterator.map(_.errorMessage).mkString("\n")))
    }
  }
}

The validate method can only return a Future, and futures completed exceptionally can only wrap an exception.

Yes you can use spray json to encode your results to json. I'm on mobile so it's difficult to provide an example right now. I'll get back to you on this when I get access to my laptop.

Thanks. An example will be very appreciated :)

Sorry if I ping you, but did you manage to create the example we talked about?

Sorry if I ping you, but did you manage to create the example we talked about?

Sorry I got really busy. I will find some time for it during this weekend.

Maybe I missed the point.

implicit class ValidationRequestMarshaller[A](um: FromRequestUnmarshaller[A]) {
  def validate(implicit validation: FormValidation[A]) = um.flatMap { _ => _ => entity =>
    validateForm(entity) {
      case Valid(_) => Future.successful(entity)
      case Invalid(failures) =>
        Future.failed(new IllegalArgumentException(
          failures.iterator.map(_.errorMessage).mkString("\n")))
    }
  }
}

The validate method can only return a Future, and futures completed exceptionally can only wrap an exception.

Hi. I just got the chance to answer this. I think I misunderstood your point before. I thought you were asking about encoding the entity to JSON (which is already what was happening in the code). I even forgot my suggestions about using SprayJson is already in the article (which only shows that time has really passed).

To answer your question, it might depend on how you want the Json to be returned. The reason we are returning an IllegalArgumentException within the future is for Akka's directive to interpret it as a validation error in the payload itself. However, you can put the stringified Json inside the message of that exception if you want.

Or you can also write your own handlers as show here: https://stackoverflow.com/questions/35165376/return-json-errors-in-akka-http-api.

Unfortunately, the latter is a very involved process.