A k-shingle is any k characters that appear consecutively in a document. If we represent a document by its set of k-shingles, then the Jaccard similarity of the shingle sets measures the textual similarity of documents. Sometimes, it is useful to hash shingles to bit strings of shorter length, and use sets of hash values to represent documents.

k-shingling-python.py

__author__ = 'renienj'


def compute_gram(doc_data, k=2):
    """
    In natural language processing a w-shingling is a set of unique "shingles"
    (n-grams, contiguous subsequences of tokens in a document)

    Very much similar to n-grams but here we consider characters
    """
    sh = set()
    if len(doc_data) >= k:
        for pos, token in enumerate(doc_data):
            if pos + k <= len(doc):
                sh.add(doc[pos:pos + k])
        return sh
    else:
        print 'Tokens are not available'
    pass


if __name__ == "__main__":

    # Documents
    documents = ["abcdabd"]

    # Shingling size
    w = 2

    for index, doc in enumerate(documents):
        print "Then the set of %s-shingles for doc[%s] : %s" % (w, index+1, compute_gram(doc_data=doc, k=w))

amazing thank you!!