Steps to generate Dgraphs fulltext index tokens

generate_fulltext_token.md

1. Install Go

2. Download Dgraph repo using go get

   go get -u github.com/dgraph-io/dgraph
   

3. Go the tok package directory inside Dgraph's source.

   cd $GOPATH/src/github.com/dgraph-io/dgraph/tok
   

4. Turn the Go modules on

   export GO111MODULE=on
   

5. Copy the following test into the tok_test.go file.

   package tok
   
   import (
           "testing"
           "github.com/stretchr/testify/require"
   )
   
   
   func TestGetTokensForFullText(t *testing.T) {
           val := "Let's Go and catch @francesc at @Gopherpalooza today, as he scans into Go source code by building its Graph in Dgraph!\nBe there, as he Goes through analyzing Go source code, using a Go program, that stores data in the GraphDB built in Go!\n#golang #GraphDB #Databases #Dgraph"
            // Call the fulltext tokenizer.
           tokens, err := (&FullTextTokenizer{lang: "en"}).Tokens(val)
           require.NoError(t, err)
           // Print the generated tokens.
           t.Logf("%v",tokens)
   } 

6. Now, run the test, this should print the fulltext tokens for the matched tweet. Change the value of the val variable in the above test to find the fulltext tokens for a diffrent sentence or a text paragraph.

   go test -run=TestGetTokensForFullText -v