Created
October 10, 2020 09:45
-
-
Save joshua-taylor/1ec22f976826f707c057b2d2f1deeefb to your computer and use it in GitHub Desktop.
sPacy tokenize
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| nlp = spacy.load("en_core_web_sm") | |
| tok_text=[] # OUTPUT for our tokenised corpus | |
| text = df.text.str.lower().values | |
| text = [fix_text(str(i)) for i in text] | |
| #Tokenising using SpaCy: | |
| for doc in tqdm(nlp.pipe(text, n_threads=2, disable=["tagger", "parser","ner"])): | |
| tok = [t.text for t in doc if (t.is_ascii and not t.is_punct and not t.is_space)] | |
| tok_text.append(tok) |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hi Alex - yes that is right it is just the standard function from ftfy - it can be removed with no impact to the rest of the code if you do not have any text encoding issues.