[Golang stream new-line delimited JSON to Server] #go #golang #api #stream #json #ndjson

0. README.md

The application/x-ndjson MIME type refers to a type of data format called Newline delimited JSON (NDJSON). NDJSON is a way of storing structured data as a sequence of JSON (JavaScript Object Notation) objects, separated by newline characters.

Each line of an NDJSON file contains a complete JSON object, which means that a single NDJSON file can contain multiple JSON objects. This format is commonly used for streaming large datasets, as it allows data to be processed in a continuous and efficient manner.

The application/x-ndjson MIME type is used to indicate that a file or data stream contains NDJSON formatted data. The x in the MIME type indicates that it is an unregistered type, which means that it is not an official MIME type defined by the Internet Assigned Numbers Authority (IANA). However, it is widely used and supported by many applications and APIs that deal with structured data.

Here's an example of how you could use Go to send an NDJSON-formatted payload to an API:

package main

import (
	"bytes"
	"encoding/json"
	"fmt"
	"net/http"
)

func main() {
	// Define some example JSON objects
	obj1 := map[string]string{"name": "Alice", "age": "30"}
	obj2 := map[string]string{"name": "Bob", "age": "40"}

	// Create a buffer to hold the NDJSON payload
	var payload bytes.Buffer

	// Encode each object as a JSON string and write it to the payload buffer with a newline separator
	enc := json.NewEncoder(&payload)
	enc.Encode(obj1)
	payload.WriteByte('\n')
	enc.Encode(obj2)
	payload.WriteByte('\n')

	// Create a new HTTP request to the API endpoint
	req, err := http.NewRequest("POST", "http://example.com/api", &payload)
	if err != nil {
		fmt.Println("Error creating request:", err)
		return
	}

	// Set the Content-Type header to indicate that we're sending NDJSON data
	req.Header.Set("Content-Type", "application/x-ndjson")

	// Send the request and print the response
	client := &http.Client{}
	resp, err := client.Do(req)
	if err != nil {
		fmt.Println("Error sending request:", err)
		return
	}
	defer resp.Body.Close()

	fmt.Println("Response status:", resp.Status)
}

In this example, we define two JSON objects (obj1 and obj2) and encode them as strings using the json package. We then write each encoded object to a buffer, with a newline character separating them. This creates an NDJSON-formatted payload.

We then create a new HTTP request to an API endpoint and set the request body to our NDJSON payload. We also set the Content-Type header to indicate that we're sending NDJSON data.

Finally, we send the request using an http.Client and print the response status. Note that this example is simplified and doesn't include error handling for the sake of brevity. In a real-world application, you would want to handle errors appropriately.

1. Example.sh

./stream.sh 100000 | go run client.go

2. stream.sh

#!/bin/bash

generate_json_object() {
  echo "{\"key\":\"$(openssl rand -hex 16)-$RANDOM-$(date +%s)\",\"value\":\"$(echo object_$RANDOM | base64)\"}"
}

for _ in $(seq 1 "$1"); do
  generate_json_object
done

3. client.go

package main

import (
	"log"
	"net/http"
	"os"
)

func main() {
	req, err := http.NewRequest("POST", "http://localhost:8080/stream", os.Stdin)
	if err != nil {
		log.Println("Failed to create HTTP request:", err)
	}
	req.Header.Set("Content-Type", "application/json")

	client := &http.Client{}
	resp, err := client.Do(req)
	if err != nil {
		log.Println("Failed to send HTTP request:", err)
	}
	defer resp.Body.Close()

	log.Println("Response:", resp.Status)
}

4. server.go

package main

import (
	"bufio"
	"encoding/json"
	"io"
	"log"
	"net/http"
	"os"
)

type MyServerObject struct {
	ID   int    `json:"id"`
	Name string `json:"name"`
}

func handleJSONStream(w http.ResponseWriter, r *http.Request) {
	reader := bufio.NewReader(io.TeeReader(r.Body, os.Stdout))
	defer r.Body.Close()

	decoder := json.NewDecoder(reader)

	for {
		var obj MyServerObject
		err := decoder.Decode(&obj)
		if err != nil {
			if err.Error() == "EOF" {
				break
			}
			http.Error(w, "Failed to decode JSON object", http.StatusBadRequest)
			return
		}

		log.Printf("Received object: %+v", obj)
	}

	w.WriteHeader(http.StatusOK)
	w.Write([]byte("Data processed successfully"))
}

func main() {
	http.HandleFunc("/stream", handleJSONStream)
	log.Fatal(http.ListenAndServe(":8080", nil))
}