Created
June 28, 2014 23:09
-
-
Save Rhomboid/9fbdf5cfb28bb90dd35b to your computer and use it in GitHub Desktop.
WebAdvisor scraping example
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import requests | |
| from bs4 import BeautifulSoup | |
| from itertools import chain | |
| baseurl = 'https://webadv.sunyrockland.edu/WebAdvisor/WebAdvisor' | |
| main_page = {'type': 'M', 'pid': 'CORE-WBMAIN'} | |
| section_search = {'CONSTITUENCY': 'WBAP', 'type': 'P', 'pid': 'ST-WESTS12A'} | |
| s = requests.Session() | |
| s.params = {'TOKENIDX': ''} | |
| r = s.get(baseurl, params=main_page) | |
| s.params['TOKENIDX'] = r.cookies['LASTTOKEN'] | |
| r = s.get(baseurl, params=main_page) | |
| r = s.get(baseurl, params=section_search) | |
| r2 = s.post(r.request.url, data={ | |
| 'VAR1': '2014/FA', | |
| 'LIST.VAR1_CONTROLLER': 'LIST.VAR1', | |
| 'LIST.VAR1_MEMBERS': 'LIST.VAR1*LIST.VAR2*LIST.VAR3*LIST.VAR4', | |
| 'LIST.VAR1_MAX': '5', | |
| 'LIST.VAR1_1': 'CSP', | |
| }) | |
| # assert 'Intro Programming' in r2.text | |
| soup = BeautifulSoup(r2.text) | |
| rows = soup.select('table[summary="Sections"] tr') | |
| labels = [th.get_text(strip=True) for th in rows[1].find_all('th')[1:]] | |
| course_data = [[td.get_text(strip=True) for td in row.find_all('td')[1:]] for row in rows[2:]] | |
| col_widths = [max(map(len, chain((label,), (course[col] for course in course_data)))) for col, label in enumerate(labels)] | |
| fmt = '|'.join('{{:{}}}'.format(w) for w in col_widths).format | |
| with open('output.txt', 'w') as output: | |
| print(fmt(*labels), file=output) | |
| print('-' * (sum(col_widths) + len(col_widths) - 1), file=output) | |
| for course in course_data: | |
| print(fmt(*course), file=output) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Term |Status|Section Name and Title |Location |Meeting Information |Faculty |Available/ Capacity|Credits|CEUs|Academic Level | |
| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | |
| Fall 2014|Open |CSP*100*001N (17513) ESL Computing |Main Campus|09/02/2014-12/18/2014 Lecture Tuesday, Thursday 11:00AM - 12:15PM, Technology Building, Room 8172 |C. Davis |16 / 20 |3.00 | |Credit | |
| Fall 2014|Open |CSP*100*070N (23605) ESL Computing |Main Campus|09/02/2014-12/18/2014 Lecture Tuesday, Thursday 05:00PM - 06:15PM, Technology Building, Room 8172 |C. Davis |15 / 15 |3.00 | |Credit | |
| Fall 2014|Open |CSP*101*001S (22984) Computer Concepts/Applications| |09/02/2014-12/22/2014 Asynchronous Online Days to be Announced, Times to be AnnouncedOnline, Room SLN |C. Kaplan-Newmark|17 / 20 |3.00 | |Credit | |
| Fall 2014|Open |CSP*101*002 (0314) Computer Concepts/Applications |Main Campus|09/02/2014-12/22/2014 Lecture Monday, Wednesday, Friday 09:00AM - 09:50AM, Technology Building, Room 8375|X. Ye |8 / 20 |3.00 | |Credit | |
| Fall 2014|Open |CSP*101*003 (21627) Computer Concepts/Applications |Main Campus|09/03/2014-12/22/2014 Lecture Monday, Wednesday, Friday 11:00AM - 11:50AM, Technology Building, Room 8174|X. Ye |17 / 20 |3.00 | |Credit | |
| Fall 2014|Open |CSP*101*005 (0303) Computer Concepts/Applications |Main Campus|09/02/2014-12/22/2014 Lecture Monday, Wednesday, Friday 12:00PM - 12:50PM, Technology Building, Room 8175|To be Announced |10 / 20 |3.00 | |Credit | |
| Fall 2014|Open |CSP*101*006 (12443) Computer Concepts/Applications |Main Campus|09/02/2014-12/22/2014 Lecture Monday, Wednesday, Friday 01:00PM - 01:50PM, Technology Building, Room 8375|X. Ye |9 / 20 |3.00 | |Credit | |
| Fall 2014|Open |CSP*101*007 (12445) Computer Concepts/Applications |Main Campus|09/02/2014-12/22/2014 Lecture Monday, Wednesday 03:00PM - 04:15PM, Technology Building, Room 8175 |To be Announced |17 / 20 |3.00 | |Credit | |
| Fall 2014|Open |CSP*101*008 (0307) Computer Concepts/Applications |Main Campus|09/02/2014-12/22/2014 Lecture Tuesday, Thursday 08:00AM - 09:15AM, Technology Building, Room 8175 |To be Announced |17 / 20 |3.00 | |Credit | |
| Fall 2014|Open |CSP*101*009 (0308) Computer Concepts/Applications |Main Campus|09/02/2014-12/22/2014 Lecture Tuesday, Thursday 09:30AM - 10:45AM, Technology Building, Room 8375 |S. Kastens |9 / 20 |3.00 | |Credit | |
| Fall 2014|Open |CSP*101*010 (21628) Computer Concepts/Applications |Main Campus|09/02/2014-12/22/2014 Lecture Monday, Wednesday, Friday 10:00AM - 10:50AM, Technology Building, Room 8375|X. Ye |15 / 20 |3.00 | |Credit | |
| Fall 2014|Open |CSP*101*011 (10373) Computer Concepts/Applications |Main Campus|09/02/2014-12/22/2014 Lecture Tuesday, Thursday 11:00AM - 12:15PM, Technology Building, Room 8173 |C. Kaplan-Newmark|11 / 20 |3.00 | |Credit | |
| Fall 2014|Open |CSP*101*012 (0311) Computer Concepts/Applications |Main Campus|09/02/2014-12/22/2014 Lecture Tuesday, Thursday 01:30PM - 02:45PM, Technology Building, Room 8375 |S. Ritt |12 / 20 |3.00 | |Credit | |
| Fall 2014|Open |CSP*101*013 (12393) Computer Concepts/Applications |Main Campus|09/02/2014-12/22/2014 Lecture Tuesday, Thursday 03:00PM - 04:15PM, Technology Building, Room 8375 |S. Ritt |16 / 20 |3.00 | |Credit | |
| Fall 2014|Open |CSP*101*016 (21629) Computer Concepts/Applications |Main Campus|09/02/2014-12/22/2014 Lecture Monday, Wednesday, Friday 02:00PM - 02:50PM, Technology Building, Room 8173|To be Announced |13 / 20 |3.00 | |Credit | |
| Fall 2014|Open |CSP*101*070 (0630) Computer Concepts/Applications |Main Campus|09/02/2014-12/22/2014 Lecture Monday 06:00PM - 08:40PM, Technology Building, Room 8175 |S. Kastens |18 / 20 |3.00 | |Credit | |
| Fall 2014|Open |CSP*101*071 (0631) Computer Concepts/Applications |Main Campus|09/02/2014-12/22/2014 Lecture Tuesday 04:00PM - 06:40PM, Technology Building, Room 8175 |H. Schuh |18 / 20 |3.00 | |Credit | |
| Fall 2014|Open |CSP*101*H74 (20548) Computer Concepts/Applications |Haverstraw |09/02/2014-12/22/2014 Lecture Monday 07:00PM - 09:40PM, Haverstraw, Room 118 |J. Campolongo |15 / 20 |3.00 | |Credit | |
| Fall 2014|Open |CSP*105*070 (23080) Computers in Society |Main Campus|09/04/2014-12/18/2014 Lecture Thursday 07:00PM - 09:40PM, Technology Building, Room 8171 |L. Juele |14 / 20 |3.00 | |Credit | |
| Fall 2014|Open |CSP*125*070 (0638) Intro Programming Visual Basic |Main Campus|09/08/2014-12/22/2014 Lecture Monday 06:00PM - 09:30PM, Technology Building, Room 8371 |To be Announced |7 / 20 |4.00 | |Credit |
Thank you! Exactly what I needed. I forked to customize for my needs.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Awesome, but can you also get course descriptions, teacher email, required textbooks, prerequisite courses, lists of courses required for majors, and more? WebAdvisor, to me, doesn't appear to have index views for this data, only individual pages. How could one scrape that data?