Skip to content

Instantly share code, notes, and snippets.

@posener
Last active February 23, 2024 21:35
Show Gist options
  • Star 84 You must be signed in to star a gist
  • Fork 3 You must be signed in to fork a gist
  • Save posener/330c2b08aaefdea6f900ff0543773b2e to your computer and use it in GitHub Desktop.
Save posener/330c2b08aaefdea6f900ff0543773b2e to your computer and use it in GitHub Desktop.
Why I Recommend to Avoid Using the go-kit Library

Why I Recommend to Avoid Using the go-kit Library

There is a trending 'microservice' library called go-kit. I've been using the go-kit library for a while now. The library provide a lot of convenience integrations that you might need in your service: with service discovery with Consul, distributed tracing with Zipkin, for example, and nice logic utilities such as round robin client side load balancing, and circuit breaking. It is also providing a way to implement communication layer, with support of RPC and REST.

The toolchain that the library provide is very nice, and it does try to solve a fundamental problem we have in the Go community: missing ecosystem to write microservices. I do like the approach of the package: 'take what you want', you can write your service and use certain tools given in this libary, and not use all of it. But, I recommend not to use the library server implementation for your REST microservices. (I think this recommendation will also apply for the RPC part of the library, but I don't have any experience with that).

You could read why, and I would love to hear your opinion about it, if you agree, disagree, and why.

1. Too verbose

Usually microservices expose APIs: external, which are exposed, and internal for inter-service communication.

When using the go-kit, it was very noticeable that the overhead of adding API you your service is very high. You need to add a lot of code, which is mostly copy-paste of other APIs and there are too many places to make mistakes.

To add a single simple API, you should add:

a. Function in the interface (make sense) b. Implementation (make sense) c. Endpoint factory function d. Transport function e. Request encoder, request decoder, response encoder and response decoder. f. Add the endpoint to the server g. Add the endpoint to the client.

For a quick impression of what I mean by "a lot of code" take a look at the simple example in the go-kit website

2. Hard to understand (at least, for me)

I think the main reason for this verbosity is the layer separation for the business logic, endpoint and transport, which is nice, and benefits in nice abstractions for the client-side load balancing, circuit breaking, tracing, etc.

But, it is hard to understand.

If you are using the go-kit as REST service library, I recommend to know the following ServerHTTP function by heart: Only then you truly understand how your service is expected to behave.

3. The interface{} API

When using the go-kit, your endpoints get an interface{} object and return an interface{}, error tuple. You need to explicitly write the conversion to your implementation function. Actually, your endpoint factory will almost be a copy-paste of the following function:

func makeUppercaseEndpoint(svc StringService) endpoint.Endpoint {
  return func(ctx context.Context, request interface{}) (interface{}, error) {
    req := request.(myRequest)
    v, err := svc.Function(req.A, req.B)
    if err != nil {
      return myResponse{v, err.Error()}, nil
    }
    return myResponse{v, ""}, nil
  }
}

To summarize

This library is really nice, and with really good intensions. It looks very popular, but can't really understand how much production ready it is, and how many actual use cases there are out there. I personally don't like the basic concepts of it, specially because of the interface{} and the verbosity issues discussed above, and I find them not easy to use, and not intuitive.

You couldn't use the nice features of the library of load-balancing, circuit breaking, and tracing, if you are not using the endpoint APIs, but if you look at the code, it is not that big. You could use the actual code or find other libraries that provide them as middleware for the standard http client/server.

Thanks for reading,

If you find this interesting, please, ✎ comment below or ★ star above ☺

For more stuff: gist.github.com/posener.

Cheers, Eyal

@rcholic
Copy link

rcholic commented Nov 21, 2017

I agree with you on the verbosity part: when I tried to understand the examples, the boilerplate code such as endpoint, request/response encoders and decoders, and the transportation layer threw me off. It is supposed to be a microservice API, why do I need to repeat these boilerplates everywhere? This kind of verbosity makes it easy to make mistakes and may be hard to maintain down the road. This being said, I think the library has some very cool concepts that help me learn go 👍

@fteem
Copy link

fteem commented Jan 18, 2018

Nice writeup! I've been looking at various microservices frameworks for Go. My idea is to basically build like a small set of micromicroservices so I can connect them and play. What alternative would you suggest for go-kit? Or, which one have you found less annoying to use?

@mdiesen
Copy link

mdiesen commented Jan 20, 2018

Nice summary. I do agree that go kit requires some boilerplate code but I like the separation of the different layers once you get your head around their purpose. When investigating different libraries I first set my eyes on goadesign which auto generates a lot of boilerplate code for you given a specification. The downside is that the DSL for the spec is fairly verbose in itself and compared to go kit I find their separation of concern less clear.
I think the best solution would be to combine the strengths of these two libraries and I will do my own attempt at that by creating templates to generate the boilerplate code required in go git (ie routing, endpints, transport, incl "hiding" the ugly interface{} and casts). If someone is interrested I'm happy to share that code once its ready.

@joejarlett
Copy link

Hey @mdiesen, did you progress with your template. I am setting out with goa v2 plus go-kit combined with the jsonapi.org spec. So keen to continue any work you have done.

@sagikazarmark
Copy link

It's hard to argue with your assessment, because every information you wrote about go-kit is true. It certainly has some solutions that come at a price, but I believe it is subjective whether you are willing or are able to pay that or not. It does not make the solutions themselves bad.

(And I don't think it was your intention to say, I just want to express that although you are absolutely right about your opinion, go-kit is not nearly as "bad" from a professional point of view as other, obviously bad solutions 😄 )

@weitzj
Copy link

weitzj commented May 8, 2018

The idea, which I kind of like is to define the services in a protobuf file (as you would do for gRPC).

This project does this really nicely. https://github.com/moul/protoc-gen-gotemplate
And the examples show some go-kit integration:

@zak905
Copy link

zak905 commented May 29, 2018

Hi @posener, thanks for the information. How complex is testing go-kit based services ? Not that simple I suppose

@hgkmail
Copy link

hgkmail commented Aug 17, 2018

go-kit need a code generator, like go-micro.

@cjyar
Copy link

cjyar commented Sep 6, 2018

@hgkmail is right. Either a code generator or use reflection to eliminate all that boilerplate code. (I'd prefer reflection.)

@jeroenvand
Copy link

I've actually started on a code generator for go-kit that generates endpoints and handles most of the boilerplate for you, starting from a Go interface definition. It's still in the WorksForMe(tm) phase, but if you're interested see: https://github.com/jeroenvand/kitboiler.

@jeroenvand
Copy link

Or perhaps have a look at this: https://github.com/kujtimiihoxha/kit - seems actually quite a bit further along and more complete

@mashingan
Copy link

Same here, couldn't comprehend the service that should be "micro" but go-kit made it so bloating in development 😞

In the end I'm doing what @jeroenvand did by writing a template for code generation. But I'm writing it in Nim 😅

@WebspressoCo
Copy link

WebspressoCo commented Dec 5, 2018

I recently compared go microservice toolkits, and the verbosity of go-kit caught my eye. Fortunately the community has been working on this for a while and now go-kit has great boilerplate generation through the cli (track this pull request for details: go-kit/kit#589)

@Henry75m39
Copy link

so, how about go-micro ?

@reyvababtista
Copy link

I do agree on these opinions of yours, @posener. Go-kit really has a lot of boilerplate codes. However, I did learn a lot about concepts and ideas of microservices and Golang by developing using the base of Go-kit. There is also another way of writing microservices in Golang using Go-micro toolkit. Go-micro declares itself as an opinionated microservices toolkit/framework. That being said, if we want to get a better grip of what microservices are, how does it works (layer by layer), how to develop one and have freedom doing it, I would still recommend to write microservices with the help of Go-kit beforehand rather than another toolkit/framework available as for now.

@ravlio
Copy link

ravlio commented Nov 13, 2019

Great article. Must read to avoid using of go-kit. This framework is nothing more than code hell. For 1 line of code, you must write 10-20 lines of boilerplate. Just check any of the examples, for instance, https://github.com/go-kit/kit/tree/master/examples/profilesvc Why you need all these Endpoints and Middlewares? Why just not to use a proxy pattern? Why just not to write straight code without func(ctx context.Context, request interface{}) (response interface{}, err error)? Use interface{} only in middlewares, in other cases you can fully control your types.

I'm 99% sure that you want to use gRPC with microservices — in that case, you absolutely do not need complex abstractions that go-kit provides. Just use gRPC directly. With its interceptors, logging, tracing, service discovery ecosystem. Straightforward, simple to debug and understand.

Another unresolvable problem with Go-kit is error passing: you can’t just take your custom error and throw it through metadata from server to client to handle it correctly. Go-kit popularizes anti-patterns like error throwing via struct properties. But it can be done easily with custom gRPC interceptors and few lines of code.

Using event-driven approach? Try to use native libraries with tiny helpers for metadata, encoding/decoding, logging, etc.. You do not need to bind your hands with go-kit thinking how to make this or that task — just do it with as little code as possible.

I'm glad that I refused to work with go-kit a long time ago. In retrospect, it was like — one day for actual business logic programming and another 2-3 days for annoying wrapping all together into endpoints and middlewares.

BTW Go-Micro looks much better designed. If you still want to stay with some framework, give go-micro a try.

Sorry if rude, but I'm upset with that fact that go-kit spread among many companies and I periodically have to work with it. Anti-go-way for sure.

@tranphuoctien
Copy link

Hi friend,
I do agree with these opinions of yours. But to understand it's you must have base knowledge things:

  • Protobuf how is working?
  • Interface in Golang? what it's?
  • You should understand how RPC send request and response!

@tomilchik
Copy link

Disclosure: I have no experience with other Golang frameworks/kits/libraries mentioned here, except go-kit.
In my work, it definitely paid for itself.
The main benefits, for my projects:

  • clean, layered separation of concerns;
  • ability to keep transport-level code (anything that has to do with http, MoM like AWS sqs/sns, tracing, db transaction management) firmly out of the way of business logic domain;
  • no-brain-involved, templated way of doing things that shouldnt really need any thinking. Think “I have an operation that I want to expose as http endpoint; of course, fully traceable, with anti-fragility built in, with audit trail logging, and db transaction management”.

Re: productivity: I timed myself, out of pure interest. Adding another http/sqs endpoint to my service’s API took between 5-10 min when I use existing data models, and up to 30-45 min when I have to create new ones. And I’m a quite slow typer :)

Where go-kit is lacking is anything to do with messaging. The amqp support is questionable - uses completely wrong (IMHO) paradigm of synchronous request/response exchange, which kinda defeats the purpose of messaging.

All in all: I’ll do a quick-n-dirty impl without go-kit, but will definitely use it for anything industrial (as in: prod-grade).

@gadieichhorn
Copy link

the idea is to have a standardized api and clean architecture design. I like go-kit as it enforces order and agreement on how things should be done in a team. Once you get your head around the concepts there is no place for hacks and breaking dependencies. When you run a large project, ordering things and creating best practice is what makes a difference.

I like the clean separation between transport and implementation. smaller code chunks do one thing and not mixing different concerns.

The middleware is a benefit when you come to construct your services, like LEGO you can add layers of instrumentation and trace without changing core services code.

making code testable is a massive benefit. injecting dependencies and abstracting interfaces is what makes it possible.

yes, if you run a very small project, it is an overkill, otherwise, I recommend it.

@RussellLuo
Copy link

I have just developed a toolkit for Go kit named kok, in order to reduce the burden of handwriting boilerplate code.

I hope you guys find it helpful :-)

@alienatorZ
Copy link

I actually switched from go-micro to go-kit for the fact that you can have complete control from transport down to the service. I got frustrated not knowing why my Java GRPC service could not call my go-micro service due to the fact that nice easy API hid all kinds of magic that was not working for me and I could not control it. Go-Kit while a LOT more verbose allows for easy debugging any issues which results in a development time savings when doing complicated orchestrations.

@eldad87
Copy link

eldad87 commented Apr 6, 2021

I wrote my own opinionated microservice example:
https://github.com/eldad87/go-boilerplate

My approach is simple, define the service HTTP & gRPC transport layers using protobuf and work your way top to bottom.
The foundations of the service are rock-solid & decoupled, which makes every component replaceable.
In addition, the HTTP layer is been document seemly using Swagger.

@ricardodovalle
Copy link

What would you recommend for microservices? @posener

@NishanthSpShetty
Copy link

anyone tried using https://github.com/go-coldbrew ?

@nyelnizy
Copy link

Why not use kubernettes with vanilla go??

@KacperPerschke
Copy link

KacperPerschke commented Jan 1, 2024

I came across prometheus/common/promlog based on go-kit/kit/log. I suggested replacing promlog with log/slog.

Usage comparison:

  1. simple case
    • promlog → level.Debug(c.logger).Log("msg", "Fetching unavailable mirrors")
    • slog → c.logger.Debug("Fetching unavailable mirrors")
  2. the case is a bit complicated
    • promlog → level.Warn(c.logger).Log("msg", "The endpoint does not exist", "endpoint", fullPath, "err", fmt.Sprintf("%v", apiErrors.Errors), "status", 404)
    • slog → I deliberately wrote it on multiple lines for readability:
c.logger.Warn(
        "The endpoint does not exist",
        "endpoint", fullPath, 
        "err", fmt.Sprintf("%v", apiErrors.Errors),
        "status", 404,
)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment