Skip to content

Instantly share code, notes, and snippets.

@monspo1

monspo1/proj3_webscrape1.py

Last active August 25, 2019 01:12
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save monspo1/f1d48a4e366c6915553c to your computer and use it in GitHub Desktop.
Save monspo1/f1d48a4e366c6915553c to your computer and use it in GitHub Desktop.
Download ZIP
Open the connection to indeed.com using BeautifulSoup
Raw
proj3_webscrape1.py
# load the library
from bs4 import BeautifulSoup as Soup
import urllib, requests, re, pandas as pd
# indeed.com url
base_url = 'http://www.indeed.com/jobs?q=data+scientist&jt=fulltime&sort='
sort_by = 'date' # sort by data
start_from = '&start=' # start page number
pd.set_option('max_colwidth',500) # to remove column limit (Otherwise, we'll lose some info)
df = pd.DataFrame() # create a new data frame
@nfeliccia
Copy link

Where's the rest of the project?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment