Skip to content

Instantly share code, notes, and snippets.

@hamletbatista

hamletbatista/crawl_delay_pe.py

Created July 25, 2019 03:53
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save hamletbatista/02fb89d885825398c611cba57fbdb3ec to your computer and use it in GitHub Desktop.
Save hamletbatista/02fb89d885825398c611cba57fbdb3ec to your computer and use it in GitHub Desktop.
Download ZIP
Raw
crawl_delay_pe.py
#first convert the dates to datetime objects
crawled["lastmod"] = pd.to_datetime(crawled["lastmod"])
crawled["date"] = pd.to_datetime(crawled["date"])
crawled["crawl_delay"] = crawled["date"] - crawled["lastmod"]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment