blog.md

TL;DR: prevailing "secondary source" wisdom (ie: blog posts) about json.Decoder don't demonstrate the proper way to use it.

---

This post is a follow up to my (kinda lengthy) deep dive into what I thought was a bug in golang's json.Decoder pkg.

Instead, I realized that generally speaking, json.Decoder can be misunderstood - which may lead to unintended consequences. In this post, I will demonstrate a safer pattern that ought to be used instead of the prevailing wisdom.

Googling: "json.decoder example golang"

I ran a few google searches queries using some permutation of the following:

json.decoder example golang

The results were your standard mix of documentation from golang.org, blog posts on medium.com/random mom&pop devsites (like this one!), and a few stackoverflow threads.

Fortunately, results from golang.org are highly ranked - while it may be a bit harder to parse through golang's src code, the documentation is fairly thorough and more importantly: correct. (Personally, I would have preferred some additional context in the docs expounding on some of the gotchas I discuss on my other post but I digress)

Some of the threads I observed in Stack Overflow that referenced json.Decoder pulled directly from the docs (and therefore were also correct). Other's (probably) pulled from medium/other blog post sites similar to the ones I found googling around and were inaccurate/advocating incorrect usage.

For example, on Medium, et al - I saw a wide array of posts (such as this, this, this or this) that suggested using json.Decoder in some way, shape, or form similarly to this:

https://gist.github.com/c58e897c47d002a7ae9c69992ba9a645

(Example pulled from Tutorial: How to work with JSON data in Go)

The Problem

On the surfact, this code looks sound. I forklifted the src into go playground and ran it.

https://gist.github.com/cbf03f8daadbcda9de5d9b3dc399948a

(Note, changed profile to map[string]interface{} to make the code run)

...It works! Great. But, what happens if we fubar the JSON string?

https://gist.github.com/1312d9726d754a75173e5e156bda08e8

(playground)

...It works!

Wait...WTF?!.

This code should not work at all! Our JSON string is clearly malformed and we expect - based on the logic - the code to panic.

This is the entire issue in a nutshell. I expound in detail in my other post but in one (kinda long) sentence:

json.Decoder.Decode was implemented for parsing streaming JSON data, meaning it will always traverse the JSON string until it finds a satisfactory, closing bracket (I use the term satisfactory here because it does use a stack to keep track of inner brackets).

So, in order to detect the malformed json, we must actually run this logic in a loop - like so:

https://gist.github.com/a07b8763a36fad0c38ac13b9ae67061d

(playground)

Note the key diff here:

https://gist.github.com/a5b6f2a9aa51e162849096616f480e37

The Fix

From the golang docs, this example says it best:

https://gist.github.com/f753b34413ac6e36567f548d55beed61

(sauce)

In this usage, we create a new json.Decoder instance from the NewDecoder method and then continually loop and try to decode a chunk of our JSON string until we detect the end of the string (sucessfully breaking out of the loop) or an error.

I would take this a step further and only prefer to use json.Decoder when I am specifically working with streaming JSON data.

At any rate, this the The Way. Please keep this in mind going forward should you choose to use json.Decoder for your JSON string parsing needs.

A few notes

• A big caveat I'd like to point out is that I did not perform any rigorous analysis of golang examples in the wild when I was inspired to write this post. My analysis comes largely from anecdotal examples I observed while researching my other deep dive article about the nature of json.scanner and json.Decoder.Decode.
• I certianly am not being critical of the medium/mom&pop blogs I linked to earlier, instead I am merely pointing out that it is very easy to assume that the trivial usage of json.Decoder.Decode presented on those posts is generally correct when in actuality this is not the case.
• I think someone (maybe me? perhaps you?) ought to open a PR against godocs to clarify gotchas/differences between json.Decoder usage vs json.Unmarshal

---

PS: this article is x-posted from my dev blog