Skip to content

Instantly share code, notes, and snippets.

Todd Hayton thayton

Block or report user

Report or block thayton

Hide content and notifications from this user.

Learn more about blocking users

Contact Support about this user’s behavior.

Learn more about reporting abuse

Report abuse
View GitHub Profile
View sheahomes.js
/*
* $ npm init -y
* $ npm install puppeteer --save
* $ node sheahomes.js
*/
const puppeteer = require('puppeteer');
const url = 'https://www.sheahomes.com/new-homes/colorado/denver-area/parker/stonewalk-at-stepping-stone/';
async function getLot(page, url) {
await page.goto(url);
View sheahomes.py
import re
import json
import urlparse
import requests
from bs4 import BeautifulSoup
class SheaHomesScraper(object):
def __init__(self):
self.url = 'https://www.sheahomes.com/new-homes/colorado/denver-area/parker/stonewalk-at-stepping-stone/'
View lennar.py
import json
import requests
class LennarScraper(object):
def __init__(self):
self.url = 'https://www.lennar.com/Services/Rest/SearchMethods.svc/GetInventoryTabDetails'
self.session = requests.Session()
self.data = {
"CommunityID":"4531",
"pageState":{
View l3com_scraper.js
/*
* Companion code for article at http://toddhayton.com/2019/02/12/revisiting-taleo-with-puppeteer/
*
* Setup:
* $ mkdir scraper/
* $ cd scraper/
View campaignfinancemd.js
/*
* Companion code for article at http://toddhayton.com/...
*
* Setup:
* $ mkdir scraper/
* $ cd scraper/
* $ npm init -y
* $ npm install puppeteer --save
* $ npm install node-fetch --save
* $ npm install progress-stream --save
View l3com_scraper.py
#!/usr/bin/env python
import json
import requests
from bs4 import BeautifulSoup
class L3ComScraper(object):
def __init__(self):
self.session = requests.Session()
View rid_scraper.js
/*
* Companion code for article at http://toddhayton.com/2018/08/01/scraping-with-puppeteer/
*
* Setup:
* $ mkdir scraper/
* $ cd scraper/
* $ npm init -y
* $ npm install puppeteer --save
*
* Usage:
View gist:76ee55882eaf26c8a0de6a1a01ef023e
#!/usr/bin/env python
import time
import json
import requests
class Scraper(object):
def __init__(self):
self.url = 'http://preowned.ferrari.com/vdata'
self.session = requests.Session()
View keybase.md

Keybase proof

I hereby claim:

  • I am thayton on github.
  • I am thayton (https://keybase.io/thayton) on keybase.
  • I have a public key ASDwUwmXo5-1N9a9CTNzTATM_EndqNJMp3O_hzTBhaLOmQo

To claim this, I am signing this object:

You can’t perform that action at this time.