ivanliu

Ref: http://developer.yahoo.com/blogs/hadoop/posts/2010/08/apache_hadoop_best_practices_a/

Apache Hadoop: Best Practices and Anti-Patterns
Wed August 18, 2010 (Updated)
by Arun C Murthy
5 Comments Bookmark Share  
Apache Hadoop is a software framework to build large-scale, shared storage and computing infrastructures. Hadoop clusters are used for a variety of research and development projects, and for a growing number of production processes at Yahoo!, EBay, Facebook, LinkedIn, Twitter, and other companies in the industry. It is a key component in several business critical endeavors representing a very significant investment and technology component. Thus, appropriate usage of the clusters and Hadoop is critical in ensuring that we reap the best possible return on this investment.

This blog post represents compendium of best practices for applications running on Apache Hadoop. In fact, we introduce the notion of aGrid Pattern which, similar to a Design Pattern, represents a general reusable solution for app

ivanliu

1. Making a data science blog
https://www.dataquest.io/blog/how-to-setup-a-data-science-blog/

2. 28 Jupyter Notebook tips, tricks and shortcuts
https://www.dataquest.io/blog/jupyter-notebook-tips-tricks-shortcuts/

3. Docker: Data Science Environment with Jupyter  (DONE)
https://www.dataquest.io/blog/docker-data-science/

4. Unsupervised Learning: Clustering

ivanliu

= System Requirements = 
+ Finacial Data Store
- Be able to store raw web pages (unstructured data) as well as extracted data (structured data)
- Use MySql for now

+ Crawler
- Crawl selected website and store raw web pages; 
- Extract fields of interest from the web page; 
- Use Python Scrapy 

ivanliu

https://www.quora.com/What-is-the-best-API-to-create-a-chatbot-in

https://developer.pandorabots.com/
https://playground.pandorabots.com/en/
https://api.ai/
http://chatscript.sourceforge.net/
https://messengerplatform.fb.com/
http://rebot.me/page/about

ivanliu

1) Introduction to PDF and good summary for PDF extraction
https://how-to.usopendata.org/en/latest/The-Basics-of-Open-Data/Working-with-PDFs/

2) Zamzar API
Test
100 conversions remaining this month
100 test conversions remaining this month
springforward
24c0966e31d200ef3981dd22def1e7811061bcb4

ivanliu

1. Spynner
https://github.com/makinacorpus/spynner

a) Install libpng
http://ethan.tira-thompson.com/Mac_OS_X_Ports.html
b) 

2. Mechanize
http://www.pythonforbeginners.com/mechanize/browsing-in-python-with-mechanize/
http://www.pythonforbeginners.com/cheatsheet/python-mechanize-cheat-sheet

ivanliu

from flask import Flask, render_template

app = Flask(__name__)

@app.route('/')
@app.route('/index')
def index(chartID = 'chart_ID', chart_type = 'bar', chart_height = 350):
	chart = {"renderTo": chartID, "type": chart_type, "height": chart_height,}
	series = [{"name": 'Label1', "data": [1,2,3]}, {"name": 'Label2', "data": [4, 5, 6]}]
	title = {"text": 'My Title'}

ivanliu

1. The Future of Investing? AI-Run Hedge Funds
https://futurism.com/the-future-of-investing-ai-run-hedge-funds/
aidyia, sentient, rebellionresearch

ivanliu

http://jtushman.github.io/blog/2013/06/17/sharing-code-across-applications-with-python/#3
https://hynek.me/articles/sharing-your-labor-of-love-pypi-quick-and-dirty/
http://python-notes.curiousefficiency.org/en/latest/index.html

ivanliu

1) A live python demo
http://pyvideo.org/pycon-us-2015/python-concurrency-from-the-ground-up-live.html
code -> https://github.com/ivanliu/concurrencylive