Zhiyang Guo jasongzy
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #!/usr/bin/python3 | |
| import datetime | |
| import re | |
| from time import sleep | |
| import PyRSS2Gen | |
| from bs4 import BeautifulSoup | |
| from selenium import webdriver | |
| # url = "http://dgxg.njust.edu.cn/_t689/main.htm" |
jasongzy
/ mercury_parse.py
Last active
February 22, 2019 16:35
This uses the Mercury parser (https://mercury.postlight.com/web-parser/) and other tools to convert an article from URL to an HTML file. Image URLs are converted into inline Base-64 encoded images. You can run the command with the following code: ./mercury_parse.py --url=<URL> --htmlfile=<htmlfile>. <url> has to exist, and <htmlfile> has to end …
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #!/usr/bin/env python | |
| import os, sys, requests, base64, urllib.request, codecs | |
| from bs4 import BeautifulSoup | |
| from optparse import OptionParser | |
| _apiKey = 'nGc0ya2J7z2aalFrGa8Gx3Q1o8grGFsn3cz58EJy' | |
| def get_content_to_file( url, htmlfilename ): | |
| with requests.Session( ) as s: |