Previously our warmup and health checks were too coupled. Our load balancers make an http call to /healthcheck on a regular interval. This is to see if it should send traffic to the application or not. It will also kill the container if it's unhealthy after so long.
This was a problem because say mongo is down for an extended time it would take down the entire microservice. The microservice would continously try to launch containers, which would then fail healthchecks during warmup, and get killed again.
Most the application might still be in a good state though but because one dependancy is down, the whole thing is down.
So, we're splitting them out into multiple health checks.
- Unhealthy State: The application isn't ready to take traffic yet, it's working on starting up. (returns 503 to ALB)
- Healthy State: The application is ready to take traffic and warmed up. (returns 200 to ALB)
- Degraded State: The applicaton is ready to take traffic but warm up failed so it'll be slow for a bit. (returns 200 to ALB)
- If application start-up fails this may never go healthy.
- Healthy State: As long as the application is listening for traffic it returns healthy, otherwise you'd get a requst timeout.
- I don't have a concrete plan on how we'd use this for sure, maybe to verify communication between two microservices. Setting up security groups and stuff I've many times wanted a way to verify that I can reach the application without having to figure out a path to hit.
- It was in the sample app I looked at and was just a couple lines of code to add it, we might find a use in the future.
- Unhealthy State: The application is in a state where requests are likely to fail.
- Healthy State: The application is preforming as expected.
- Degraded State: The application is in a state where requests might be delayed or sane functionality is disabled. We should investigate but users should be fine.
- We won't have anything reading
Warm-up isn't a health check itself, but the in progress/complete state of the warm up is used to determine the result of the 'readiness' healthcheck.
For now you'll need to use the marvel.platform branch I've created which has the new warmup/healthcheck stuff.
dotnet add package Hudl.Marvel.Platform --version 28.5.1-MARVEL2479heal48d016
With the move to the new warm up and Health checks we're moving to ASP.NET Core 2.2. However we're keeping .NET Core at 2.1. You may need to update some ASP.NET library versions if they're used by adapters.
Previously for adding adapter packages to services, we were using reflection and this 'AddOn' functionality. This made it so it wasn't clear on what the application was using and reflection is also slow. So we now need to register the adapter packages with DI. To make it easier we're using this ApplicationBuilderExtensions and ServiceCollectionExtensions.
The ServiceCollectorExtensions configures DI and adds services for the adapter. Typically are named AddServiceName().
The ApplicationBuilderExtensions is used to configure an application's request pipeline. Typically these are named UseServiceName().
An adapter might use both of these or just one depending on what all needs to be configured.
You may see that I'm putting these in a Microsoft.Extensions.DependencyInjection namespace which might seem weird since that's outside the library, however it makes it easier to find the UseBlah() and AddBlah() methods and reduces the amount of 'usings' clutter like in here https://github.com/hudl/dotnet-platform/blob/master/src/Hudl.Marvel.Platform/Startup/PlatformServiceCollectionExtensions.cs#L5
UPDATE: After talking with Jared we decided to use: Hudl.Marvel.DependencyInjection instead.
You can define multiple checks for a healthcheck. Each check will run on each healthcheck call.
So Mongo connectivity might be a check. Eureka connectivity might be another check.
These checks are combined into a healthcheck. If the a single check's result is unhealthy, the whole healthcheck result is unhealthy.
You'll need to add a reference to Microsoft.AspNetCore.Diagnostics.HealthChecks version 2.2.0 and the check must implement IHealthCheck
Example Healthcheck: https://github.com/hudl/dotnet-rabbitmq-adapter/blob/a84c0b17886cf5fd0326c7f0f126d9c1de0a26bd/src/Hudl.Marvel.RabbitMq.Adapter/RabbitHealthCheck.cs
I don't have a way yet to add checks from an adapter, but it'll look similar to this. https://github.com/hudl/dotnet-platform/blob/6bf03dfc8b82743386be89d8d38430db2e6dce7c/src/Hudl.Marvel.Platform/Startup/PlatformServiceCollectionExtensions.cs#L68-L69 I'll be working on that tomorrow.
Warm ups must implement IWarmUpStep, which will have 1 function you need to implement, along with a WarmUpLifecycleStep Property that you must set to either Initialization, Warm, or OnReady. Depending on which you choose, these steps will be ran at the appropriate time.
-
Initialization - Work needed to get the provider in a state that it can work, but it's not yet handling traffic. Ready but not in use.
- You won't know the order initialzations are processed, so if adapters rely on each other do that work in Warm() or OnReady because all adapters are initialized then.
-
Warm - Work to warm the adapter so the first requests are not slow and users don't have poor performance after a deploy. create connections, etc. Load inital like hudl-platform loading it's csv file.
-
OnReady - Start doing the work. Ex: begin consuming from rabbit, janus.
Example of registering warm up within adapter: https://github.com/hudl/dotnet-rabbitmq-adapter/blob/a84c0b17886cf5fd0326c7f0f126d9c1de0a26bd/src/Hudl.Marvel.RabbitMq.Adapter/DependencyInjection/ServiceCollectionExtensions.cs#L108
Example of registering adapter with microservice application builder: https://github.com/hudl/hudl-microservicetemplate/blob/e7fa619a73d6876a2e3d772666d3047f59792a21/src/Hudl.MicroserviceTemplate.Webapp/Startup.cs#L44-L45
Example of registering adapter with microservice with ServieCollection: https://github.com/hudl/hudl-microservicetemplate/blob/MARVEL-2479-healthcheck-and-warmup/src/Hudl.MicroserviceTemplate.Webapp/Startup.cs#L22-L23
Platform Changes: https://github.com/hudl/dotnet-platform/pull/225 RabbitMq Adapter example: https://github.com/hudl/dotnet-rabbitmq-adapter/tree/MARVEL-2481-new-warmup Microservice Template Example: https://github.com/hudl/hudl-microservicetemplate/pull/87
Great article but all links are broken unfortunatly