Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Steps to generate Dgraphs fulltext index tokens
  1. Install Go

  2. Download Dgraph repo using go get

    go get -u github.com/dgraph-io/dgraph
    
  3. Go the tok package directory inside Dgraph's source.

    cd $GOPATH/src/github.com/dgraph-io/dgraph/tok
    
  4. Turn the Go modules on

    export GO111MODULE=on
    
  5. Copy the following test into the tok_test.go file.

    package tok
    
    import (
            "testing"
            "github.com/stretchr/testify/require"
    )
    
    
    func TestGetTokensForFullText(t *testing.T) {
            val := "Let's Go and catch @francesc at @Gopherpalooza today, as he scans into Go source code by building its Graph in Dgraph!\nBe there, as he Goes through analyzing Go source code, using a Go program, that stores data in the GraphDB built in Go!\n#golang #GraphDB #Databases #Dgraph"
             // Call the fulltext tokenizer.
            tokens, err := (&FullTextTokenizer{lang: "en"}).Tokens(val)
            require.NoError(t, err)
            // Print the generated tokens.
            t.Logf("%v",tokens)
    } 
  6. Now, run the test, this should print the fulltext tokens for the matched tweet. Change the value of the val variable in the above test to find the fulltext tokens for a diffrent sentence or a text paragraph.

    go test -run=TestGetTokensForFullText -v
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.