matt-peters

"""
An example of running both pytorch and tensorflow in the same network,
while pasing weights and gradients between the two.

In this example, we run a simple 2-layer feed-forward network,
with the first layer size (5, 2) and the second (2, 3).
The code contains an implementation of forward/backward passes with
three versions:
    * tensorflow only
    * pytorch only

matt-peters

'''
data from: http://lcl.uniroma1.it/wsdeval/

(0.1) Run convert_all_wsd_datasets to convert to JsonFile
(0.2) run remove_semeval2007_gold to remove semeval2007 from the gold data

(1) dump embeddings for the datasets to hdf5 file

(2) Compute the average word embedding for each sense in the training data
    with generate_sense_vectors

matt-peters

import requests
from dragnet.models import content_extractor


u = 'https://github.com/seomoz/dragnet'
html = requests.get(u).content

blocks = content_extractor.analyze(html, blocks=True)

block_text = [block.text for block in blocks]

matt-peters

from dragnet import content_extractor, content_comments_extractor

with open('msn_blockifyerror.txt', 'r') as fin:
    html = fin.read()

content = content_extractor.analyze(html, encoding='utf-8')
content_comments = content_comments_extractor.analyze(html, encoding='utf-8')

print(content)