Turn data into Go structs - Martin Czygan

Golab.md

Turn data into Go structs

• https://www.golab.io/, 2018-10-22, 18:00

---

Tools

Go encourages tools: gofmt, gorename, stringer and many more.

---

Go uses struct tags

Go offers struct tags which are discoverable via reflection. These enjoy a wide range of use in the standard library in the JSON/XML and other encoding packages.

---

JSON and XML

When working with data, JSON and XML are common serialization formats.

---

The /ˈtiːdɪəm/

Have you ever written JSON or XML struct tags by hand before?

I did. It is not hard, but can become tedious and error-prone, if the data structures become large.

---

Automation

Can we automate the step that takes us from raw JSON and XML data to a nice Go struct?

Yes, there are various options.

---

JSONGen

• https://github.com/bemasher/JSONGen

JSONGen is a tool for generating native Golang types from JSON objects.

---

Reduce effort by 99%

• Replace 3208 keystrokes with 30.

$ curl -sL https://is.gd/xG6wk4 | JSONGen
$ type _ struct {
	Message struct {
		AlternativeId  []string `json:"alternative-id"`
		ContainerTitle []string `json:"container-title"`
		ContentDomain  struct {
			CrossmarkRestriction bool          `json:"crossmark-restriction"`
			Domain               []interface{} `json:"domain"`
		} `json:"content-domain"`
		Created struct {
			DateParts []int64 `json:"date-parts"`
			DateTime  string  `json:"date-time"`
			Timestamp int64   `json:"timestamp"`
		} `json:"created"`
		DOI       string
		Deposited struct {
			DateParts []int64 `json:"date-parts"`
			DateTime  string  `json:"date-time"`
			Timestamp int64   `json:"timestamp"`
		} `json:"deposited"`
		ISSN    []string
		Indexed struct {
			DateParts []int64 `json:"date-parts"`
			DateTime  string  `json:"date-time"`
			Timestamp int64   `json:"timestamp"`
		} `json:"indexed"`
		IsReferencedByCount int64 `json:"is-referenced-by-count"`
		IssnType            []struct {
			Type  string `json:"type"`
			Value string `json:"value"`
		} `json:"issn-type"`
		Issue  string `json:"issue"`
		Issued struct {
			DateParts []int64 `json:"date-parts"`
		} `json:"issued"`
		JournalIssue struct {
			Issue           string `json:"issue"`
			PublishedOnline struct {
				DateParts []int64 `json:"date-parts"`
			} `json:"published-online"`
			PublishedPrint struct {
				DateParts []int64 `json:"date-parts"`
			} `json:"published-print"`
		} `json:"journal-issue"`
		Language string `json:"language"`
		License  []struct {
			ContentVersion string `json:"content-version"`
			DelayInDays    int64  `json:"delay-in-days"`
			Start          struct {
				DateParts []int64 `json:"date-parts"`
				DateTime  string  `json:"date-time"`
				Timestamp int64   `json:"timestamp"`
			} `json:"start"`
			URL string
		} `json:"license"`
		Link []struct {
			ContentType         string `json:"content-type"`
			ContentVersion      string `json:"content-version"`
			IntendedApplication string `json:"intended-application"`
			URL                 string
		} `json:"link"`
		Member          string        `json:"member"`
		OriginalTitle   []interface{} `json:"original-title"`
		Page            string        `json:"page"`
		Prefix          string        `json:"prefix"`
		PublishedOnline struct {
			DateParts []int64 `json:"date-parts"`
		} `json:"published-online"`
		PublishedPrint struct {
			DateParts []int64 `json:"date-parts"`
		} `json:"published-print"`
		Publisher       string `json:"publisher"`
		ReferenceCount  int64  `json:"reference-count"`
		ReferencesCount int64  `json:"references-count"`
		Relation        struct {
		} `json:"relation"`
		Score               float64       `json:"score"`
		ShortContainerTitle []string      `json:"short-container-title"`
		ShortTitle          []interface{} `json:"short-title"`
		Source              string        `json:"source"`
		Subtitle            []interface{} `json:"subtitle"`
		Title               []string      `json:"title"`
		Type                string        `json:"type"`
		URL                 string
		Volume              string `json:"volume"`
	} `json:"message"`
	MessageType    string `json:"message-type"`
	MessageVersion string `json:"message-version"`
	Status         string `json:"status"`
}

---

XML

I was searching for something similar for XML, which is more complex than JSON. There are:

• https://github.com/dutchcoders/XMLGen
• https://github.com/miku/zek

Mapping between XML elements and data structures is inherently flawed: an XML element is an order-dependent collection of anonymous values, while a data structure is an order-independent collection of named values.

---

XMLGen

Similar to JSONGen, very easy to use.

$ curl -s http://www.ibiblio.org/xml/examples/shakespeare/all_well.xml | XMLGen
type _ struct {
	PLAY struct {
		TITLE string `xml:"TITLE"`
		FM    struct {
			P []string `xml:"P"`
		} `xml:"FM"`
		PERSONAE struct {
			TITLE   string   `xml:"TITLE"`
			PERSONA []string `xml:"PERSONA"`
			PGROUP  struct {
				PERSONA  []string `xml:"PERSONA"`
				GRPDESCR string   `xml:"GRPDESCR"`
			} `xml:"PGROUP"`
			PERSONA []string `xml:"PERSONA"`
			PGROUP  struct {
				PERSONA  []string `xml:"PERSONA"`
				GRPDESCR string   `xml:"GRPDESCR"`
			} `xml:"PGROUP"`
			PERSONA string `xml:"PERSONA"`
		} `xml:"PERSONAE"`
		SCNDESCR string `xml:"SCNDESCR"`
		PLAYSUBT string `xml:"PLAYSUBT"`
		ACT      []struct {
			TITLE string `xml:"TITLE"`
			SCENE []struct {
				TITLE    string `xml:"TITLE"`
				STAGEDIR string `xml:"STAGEDIR"`
				SPEECH   []struct {
					SPEAKER  string `xml:"SPEAKER"`
					LINE     string `xml:"LINE"`
					STAGEDIR string `xml:"STAGEDIR"`
				} `xml:"SPEECH"`
				STAGEDIR string `xml:"STAGEDIR"`
				SPEECH   []struct {
					SPEAKER  string   `xml:"SPEAKER"`
					LINE     []string `xml:"LINE"`
					STAGEDIR string   `xml:"STAGEDIR"`
				} `xml:"SPEECH"`
				STAGEDIR string `xml:"STAGEDIR"`
				SPEECH   []struct {
					SPEAKER  string   `xml:"SPEAKER"`
					LINE     []string `xml:"LINE"`
					STAGEDIR []string `xml:"STAGEDIR"`
					LINE     []string `xml:"LINE"`
				} `xml:"SPEECH"`
				STAGEDIR string `xml:"STAGEDIR"`
				SPEECH   struct {
					SPEAKER  string `xml:"SPEAKER"`
					LINE     string `xml:"LINE"`
					STAGEDIR string `xml:"STAGEDIR"`
				} `xml:"SPEECH"`
				STAGEDIR string `xml:"STAGEDIR"`
				SPEECH   []struct {
					SPEAKER  string   `xml:"SPEAKER"`
					LINE     []string `xml:"LINE"`
					STAGEDIR string   `xml:"STAGEDIR"`
				} `xml:"SPEECH"`
				STAGEDIR string `xml:"STAGEDIR"`
				SPEECH   struct {
					SPEAKER  string   `xml:"SPEAKER"`
					LINE     []string `xml:"LINE"`
					STAGEDIR string   `xml:"STAGEDIR"`
				} `xml:"SPEECH"`
				STAGEDIR string `xml:"STAGEDIR"`
			} `xml:"SCENE"`
			EPILOGUE struct {
				TITLE  string `xml:"TITLE"`
				SPEECH struct {
					SPEAKER string   `xml:"SPEAKER"`
					LINE    []string `xml:"LINE"`
				} `xml:"SPEECH"`
				STAGEDIR string `xml:"STAGEDIR"`
			} `xml:"EPILOGUE"`
		} `xml:"ACT"`
	} `xml:"PLAY"`
}

---

zek

While in XML, order can matter, it does not necessary matter.

Plus:

• example
• can work with multiple files

$ curl -s http://www.ibiblio.org/xml/examples/shakespeare/all_well.xml | zek -e -c
// PLAY was generated 2018-10-21 23:22:49 by tir on sol.
type PLAY struct {
        XMLName xml.Name `xml:"PLAY"`
        Text    string   `xml:",chardata"`
        TITLE   string   `xml:"TITLE"` // All's Well That Ends Well...
        FM      struct {
                Text string   `xml:",chardata"`
                P    []string `xml:"P"` // Text placed in the public...
        } `xml:"FM"`
        PERSONAE struct {
                Text    string   `xml:",chardata"`
                TITLE   string   `xml:"TITLE"`   // Dramatis Personae
                PERSONA []string `xml:"PERSONA"` // KING OF FRANCE, DUKE OF F...
                PGROUP  []struct {
                        Text     string   `xml:",chardata"`
                        PERSONA  []string `xml:"PERSONA"`  // Steward, Clown, VIOLENTA,...
                        GRPDESCR string   `xml:"GRPDESCR"` // servants to the Countess ...
                } `xml:"PGROUP"`
        } `xml:"PERSONAE"`
        SCNDESCR string `xml:"SCNDESCR"` // SCENE  Rousillon; Paris; ...
        PLAYSUBT string `xml:"PLAYSUBT"` // ALL'S WELL THAT ENDS WELL...
        ACT      []struct {
                Text  string `xml:",chardata"`
                TITLE string `xml:"TITLE"` // ACT I, ACT II, ACT III, A...
                SCENE []struct {
                        Text     string   `xml:",chardata"`
                        TITLE    string   `xml:"TITLE"`    // SCENE I.  Rousillon. The ...
                        STAGEDIR []string `xml:"STAGEDIR"` // Enter BERTRAM, the COUNTE...
                        SPEECH   []struct {
                                Text    string `xml:",chardata"`
                                SPEAKER string `xml:"SPEAKER"` // COUNTESS, BERTRAM, LAFEU,...
                                LINE    []struct {
                                        Text     string `xml:",chardata"` // In delivering my son from...
                                        STAGEDIR string `xml:"STAGEDIR"`  // To HELENA, To BERTRAM, Kn...
                                } `xml:"LINE"`
                                STAGEDIR []string `xml:"STAGEDIR"` // Enter PAROLLES, Aside, Ex...
                        } `xml:"SPEECH"`
                } `xml:"SCENE"`
                EPILOGUE struct {
                        Text   string `xml:",chardata"`
                        TITLE  string `xml:"TITLE"` // EPILOGUE
                        SPEECH struct {
                                Text    string   `xml:",chardata"`
                                SPEAKER string   `xml:"SPEAKER"` // KING
                                LINE    []string `xml:"LINE"`    // The king's a beggar, now ...
                        } `xml:"SPEECH"`
                        STAGEDIR string `xml:"STAGEDIR"` // Exeunt
                } `xml:"EPILOGUE"`
        } `xml:"ACT"`
} 

---

Tools FTW

Before you write a struct tag for existing JSON or XML data manually, you might want to try one of these utilities first.

---

Thanks

Golab_2018_LightningTalk_MartinCzygan.key.gz

Golab_2018_LightningTalk_MartinCzygan.pdf