Datastore Emulator consistency

_README.md

TLDR: The datastore emulator has a consistency setting. When developing, always set this to a value to 0.05 or lower to simulate realistic eventual consistency.

The Google Datastore is only partly consistent: only lookups and ancestor queries are strongly consistent. All types of queries use the indexes, which take a while to process in a distributed system like Google Datastore. When you develop an application that uses the Datastore, it helps to use the emulator in a way that simulates this eventual consistency. This way to can test if your application still works if the indexes are slow to update as they are in production. However, to what value should we set the consistency? It accepts a range from 0 to 1, for 0% to 100% consistency. But what does this mean?

Benchmark

See benchmark.go for a reference implementation for how this was measured. Each time the datastore emulator was started and stopped like this:

docker run --rm -d --name datastore -p 18081:18081 \
  google/cloud-sdk:emulators \
  gcloud beta emulators datastore start \
    --project=dummy --host-port=0.0.0.0:18081 \
    --consistency=CONSISTENCY_SETTING
export DATASTORE_EMULATOR_HOST=localhost:18081
go run benchmark.go
docker kill datastore

Results

The results are summarized with a boxplot of latency per --consistency setting. The x-axis is logarithmic. Only the last 3 consistency decrease steps of the test where logarithmic, but those give the strong impression that this setting is interpreted logarithmically.

Impact

Consider how these latencies affect your application. For example, an client-server application where the client makes two calls, the first a PUT to insert a new entity, and a second a GET to list all existing entities, will have issues with displaying the newly inserted entity consistently depending on network conditions & how fast the remaining parts of the client/server are.

In your localhost network the client/server communication might be so fast (1ms) that it misses the indexing even for very high consistency settings (10ms spend indexing), always appearing inconsistent. However, if the time between GET and PUT is always more than 20ms then the index will always appear consistent, and you might have surprises when deploying to production where the index update might take in 1s or more.

Conclusion

It is important to tweak your consistency setting to take into account your development and test environments properties. A safe bet is to set consistency so low that you essentially always miss the index update for anything that does not require user interaction. So use --consistency=0.05 for a latency of approximately 350ms.

Good practice would be for the aforementioned application to remain on the detail screen of the newly inserted entity. If the user navigates back to the list, substantial time will have passed and the list would refresh using the updated index. The list UI should also include a refresh button or automatically refresh periodically.

If eventual consistency is not acceptable, consider storing references in a predictable entity. For example, if a user has todo list items, store a reference to the todo list items in the user, or save the todo list items with the users as an ancestor.

References:

• Google Datastore documentation
• Google Datastore documentation > Best Practices

benchmark.go

package main

import (
	"context"
	"fmt"
	"log"
	"math/rand"
	"os"
	"strings"
	"time"

	"cloud.google.com/go/datastore"
	google_datastore "cloud.google.com/go/datastore"
)

var dsClient *google_datastore.Client
var gctx = context.Background()

func init() {
	var err error
	proj := os.Getenv("DATASTORE_PROJECT_ID")
	if proj == "" {
		proj = "dummy"
	}
	dsClient, err = google_datastore.NewClient(gctx, proj)
	orPanic(err)
}

func main() {
	// use randomness to allow testing independent updates
	random := rand.New(rand.NewSource(1337))
	times := []string{}
	for i := 0; i < 10; i++ {
		// Insert some stuff into the Datastore
		parent := datastore.IDKey("Foobar", random.Int63(), nil)
		child := datastore.IDKey("Nested", random.Int63(), parent)
		_, err := dsClient.Put(gctx, parent, &datastore.PropertyList{
			{Name: "test1", Value: parent.ID, NoIndex: false},
			{Name: "test2", Value: []interface{}{"foo", "bar"}, NoIndex: false},
			{Name: "test3", Value: []interface{}{&datastore.Entity{Key: child, Properties: []datastore.Property{
				{Name: "nested1", Value: child.ID, NoIndex: false},
			}}}, NoIndex: false},
		})
		orPanic(err)
		var start = time.Now()

		// Poll to see how soon a Query returns it
		var keys1 []*datastore.Key
		var keys2 []*datastore.Key
		for ; time.Since(start) < 10*time.Second; time.Sleep(20 * time.Millisecond) {
			keys1, err = dsClient.GetAll(gctx, datastore.NewQuery("Foobar").Filter("test1 =", parent.ID).KeysOnly().Limit(10), nil)
			orPanic(err)
			keys2, err = dsClient.GetAll(gctx, datastore.NewQuery("Foobar").Filter("test3.nested1 =", child.ID).KeysOnly().Limit(10), nil)
			orPanic(err)
			if len(keys1) > 0 && len(keys2) == 0 {
				// only one became consistent
				log.Printf("Partial consistency (parent) after: %s", time.Since(start))
			} else if len(keys1) == 0 && len(keys2) > 0 {
				// only one became consistent
				log.Printf("Partial consistency (child) after: %s", time.Since(start))
			}
			if len(keys1) > 0 && len(keys2) > 0 {
				break
			}
		}

		// Check that it resolves at some point
		d := time.Since(start)
		ms := float64(d.Microseconds()) / 1000.0
		times = append(times, fmt.Sprintf("%f", ms))
		log.Printf("Consistency after: %f ms", ms)
	}
	log.Printf("Iteration done")
	log.Printf("csv column data:\n%s", strings.Join(times, "\n"))
}

func orPanic(err error) {
	if err != nil {
		panic(err)
	}
}

consistency.png

consistency_data.csv

script.r

df <- read.csv("consistency_data.csv")
boxplot(df/1000,
        main = "Effect of --consistency",
        las = 2,
        horizontal = TRUE,
        names = c("0.9", "0.8", "0.7", "0.6","0.5", "0.4", "0.3", "0.2", "0.1", "0.05", "0.025", "0.01", "0.005"),
        xlab="time until consistent (s)",
        ylab="consistency setting",
        log = "x"
)

Super nice en interessant !! Dit is iets waar ik in ieder geval nooit aan denk wanneer ik bezig ben. Maar vanaf nu dus wel ! Echt heel tof en leerzaam. Lekker gedaan ! 💯

Now that datastore has been migrated to "Firestore in Datastore mode" which removes the eventual-consistency limitation: https://cloud.google.com/datastore/docs/firestore-or-datastore#in_datastore_mode

Eventual consistency: Datastore queries become strongly consistent unless you explicitly request [eventual consistency](https://cloud.google.com/datastore/docs/reference/data/rpc/google.datastore.v1#google.datastore.v1.ReadOptions).

Do I need to explicitly set --consistency to 0, or is this set by default?

I'm not sure @OscarVanL. Unfortunately the emulators are closed source.

@hermanbanken Thanks. As it turns out there is a flag --use-firestore-in-datastore-mode which makes reads always strongly consistent, which is most suitable for people who have been ported to Firestore in Datastore mode.

That is very goed information to document here as well. Thanks for the link 🙏