Skip to content

Instantly share code, notes, and snippets.

@agnelvishal

agnelvishal/pylinkvalidate.py

Created June 3, 2018 12:24
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save agnelvishal/7eaa9c25fe8b83ad9724be9eb23675da to your computer and use it in GitHub Desktop.
Save agnelvishal/7eaa9c25fe8b83ad9724be9eb23675da to your computer and use it in GitHub Desktop.
Download ZIP
pylinkvalidator crawler simple program. (Python 2.7)
Raw
pylinkvalidate.py
#install pylinkvalidator and lxml from pip
from pylinkvalidator.api import crawl_with_options
import lxml
crawled_site = crawl_with_options(["https://en.wikipedia.org/wiki/Introduction_to_quantum_mechanics"], {"progress":True, "parser":'lxml', "workers": 400,"depth":1000})
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment