gobinathm

Have you ever wanted to get a specific data from another website but there's no API available for it? That's where Web Scraping comes in, if the data is not made available by the website we can just scrape it from the website itself.

But before we dive in let us first define what web scraping is. According to Wikipedia:

{% blockquote %} Web scraping (web harvesting or web data extraction) is a computer software technique of extracting information from websites. Usually, such software programs simulate human exploration of the World Wide Web by either implementing low-level Hypertext Transfer Protocol (HTTP), or embedding a fully-fledged web browser, such as Internet Explorer or Mozilla Firefox. {% endblockquote %}

gobinathm

var userAgent = window.navigator.userAgent;
if (userAgent.match(/iPad/i) || userAgent.match(/iPhone/i)) {
 window.onpageshow = function(event) {
    if (event.persisted) {
        window.location.reload() 
    }
 };
}

gobinathm

// Use Gists to store code you would like to remember later on
console.log(window); // log the "window" object to the console

gobinathm

Libraries

• jQuery - The de-facto library for the modern age. It makes things like HTML document traversal and manipulation, event handling, animation, and Ajax much simpler with an easy-to-use API that works across a multitude of browsers.
• Backbone - Backbone.js gives structure to web applications by providing models with key-value binding and custom events, collections with a rich API of enumerable functions, views with declarative event handling, and connects it all to your existing API over a RESTful JSON interface.
• AngularJS - Conventions based MVC framework for HTML5 apps.
• Underscore - Underscore is a utility-belt library for JavaScript that provides a lot of the functional programming support that you would expect in Prototype.js (or Ruby), but without extending any of the built-in JavaScript objects.
• lawnchair - Key/value store adapter for indexdb, localStorage

gobinathm

Gems

• Bundler - Bundler maintains a consistent environment for ruby applications. It tracks an application's code and the rubygems it needs to run, so that an application will always have the exact gems (and versions) that it needs to run.
• rabl - General ruby templating with json, bson, xml, plist and msgpack support
• Thin - Very fast and lightweight Ruby web server
• Unicorn - Unicorn is an HTTP server for Rack applications designed to only serve fast clients on low-latency, high-bandwidth connections and take advantage of features in Unix/Unix-like kernels.
• SimpleCov - SimpleCov is a code coverage analysis tool for Ruby 1.9.
• Zeus - Zeus preloads your Rails app so that your normal development tasks such as console, server, generate, and specs/tests take less than one second.
• [factory_girl](h

gobinathm

Packages

• lxml - Pythonic binding for the C libraries libxml2 and libxslt.
• boto - Python interface to Amazon Web Services
• Django - Django is a high-level Python Web framework that encourages rapid development and clean, pragmatic design.
• Fabric - Library and command-line tool for streamlining the use of SSH for application deployment or systems administration task.
• PyMongo - Tools for working with MongoDB, and is the recommended way to work with MongoDB from Python.
• Celery - Task queue to distribute work across threads or machines.
• pytz - pytz brings the Olson tz database into Python. This library allows accurate and cross platform timezone calculations using Python 2.4 or higher.

Guides

gobinathm

Libraries

• 960 Grid System - An effort to streamline web development workflow by providing commonly used dimensions, based on a width of 960 pixels. There are two variants: 12 and 16 columns, which can be used separately or in tandem.
• Compass - Open source CSS Authoring Framework.
• Bootstrap - Sleek, intuitive, and powerful mobile first front-end framework for faster and easier web development.
• Font Awesome - The iconic font designed for Bootstrap.
• Zurb Foundation - Framework for writing responsive web sites.
• SASS - CSS extension language which allows variables, mixins and rules nesting.
• Skeleton - Boilerplate for responsive, mobile-friendly development.

Guides

gobinathm

<IfModule headers_module>
    Header set X-Content-Type-Options nosniff
</IfModule>

gobinathm

#Initial Step is to install / Step Apache
sudo apt-get install apache2

#below steps are done assuming that your website will be hosted under /var/www

#Create a directory that will act as a document root for your site
sudo mkdir -p /var/www/example.dev/public_html

#Create a directory that will used for Log Storage
sudo mkdir -p /var/www/example.dev/log

gobinathm

#Generate one time login link.
drush uli <<some-username>> 

#(Re)Set the password for the user account with the specified name.
drush user-password <<someuser>> --password="new password"