d2207197

d2207197

README

Original Data

    <record>
        <header>
           <identifier>oai:CiteSeerX.psu:10.1.1.1.1484</identifier>

d2207197

d2207197

d2207197

In [28]: paste

def count(c = []):
    c.append(len(c))
    return c
## -- End pasted text --

In [29]: count()
Out[29]: [0]

d2207197

d2207197

d2207197

steps

1. Use the text in VOA.txt for building LM
2. Build a dictionary of ngrams and ngram counts (n = 1, 2) (split by newline, lowercase )
3. Compute Nr for 1gram and 2gram
4. Compute (r, r*), for 1gram and 2gram (k=10)
5. Compute the normalization factor N / (N + k Nk)
6. Good-Turing Estimation of P(w) and P(w’|w)
7. Compute probability P(w1, w2, .., wn) of sentences (w1, w2, .., wn) on the course webpage

d2207197

d2207197