Skip to content

Instantly share code, notes, and snippets.

Avatar
😀

Tejas Vaidhya tejasvaidhyadev

😀
View GitHub Profile
@tejasvaidhyadev
tejasvaidhyadev / Sentencepiece_unigram.jl
Created Jun 26, 2020
Experimental Implementation of Sentencepiece in julia
View Sentencepiece_unigram.jl
function load(path)
vocab = readlines(path)
vocabnew = split.(vocab , "\t")
vo = []
for i in 1:30000
vocab1 = vocabnew[i][1]
vocab1 = replace(vocab1,""=>"_")
push!(vo,vocab1)
end
vocab1 = convert(Array{String,1},vo)
@tejasvaidhyadev
tejasvaidhyadev / test.jl
Last active Feb 10, 2020
For testing Avg Perceptron POS present in julia TextAnalysis.jl
View test.jl
using CorpusLoaders
using TextAnalysis
using MultiResolutionIterators
test=load(CoNLL(),"test")
test=flatten_levels(test, lvls(CoNLL, :document)) |> full_consolidate
testx=Vector{String}()
testy=Vector{String}()
tagger =PerceptronTagger(true)
for val in 1:length(test)
You can’t perform that action at this time.