Skip to content

Instantly share code, notes, and snippets.

@conredwang
conredwang / skyscraper_spider.py
Created November 11, 2016 21:09
Scraping Skyscrapers data using Scrapy. Code 3 of 3. skyscraper_spider.py
# -*- coding: utf-8 -*-
# author : Conred Wang
# title : Wed scraping using Scrapy. Code 2 of 3. skyscraper_spider.py
from scrapy import Spider
from scrapy.selector import Selector
from skyscraper.items import SkyscraperItem
import scrapy
@conredwang
conredwang / pipelines.py
Created November 11, 2016 21:06
Scraping Skyscrapers data using Scrapy. Code 2 of 3. pipelines.py
# -*- coding: utf-8 -*-
# author : Conred Wang
# title : Wed scraping using Scrapy. Code 2 of 3. pipelines.py
class WriteItemPipeline(object):
def __init__(self):
self.filename = 'tall.txt'
def open_spider(self, spider):
@conredwang
conredwang / items.py
Created November 11, 2016 21:01
Scraping Skyscrapers data using Scrapy. Code 1 of 3. items.py
-*- coding: utf-8 -*-
# author : Conred Wang
# title : Wed scraping using Scrapy. Code 1 of 3. items.py
from scrapy import Item, Field
# SkyscraperItem has information about one skyscraper/building:
# Type-1. Some fields are from primary page "https://skyscrapercenter.com/buildings".
# Type-2. Some fields are from secondary pages by navigating the skyscraper's/building's specific url.
@conredwang
conredwang / I.Told.You.So..Rmd
Created October 18, 2016 17:02
I Told You So! ...<R, gglot2, knitR>
---
title: “I Told You So!"
author: “Conred Wang“
date: “10/10/2016”
output:
ioslides_presentation:
logo: assets/img/i.told.you.so.jpg
css: assets/css/ioslides.css
abstract:
* R visualization using gglot2.