| # Install deltarpm to reduce the size of download | |
| yum install -y deltarpm \ | |
| && yum install -y python-pip \ | |
| && pip install --upgrade pip | |
| # Scrapy Pre-Requisites | |
| yum install -y gcc libffi-devel python-devel openssl-devel | |
| yum install -y libxslt-devel libxml++-devel libxml2-devel | |
| pip install lxml incremental |
https://www.youtube.com/watch?v=d0yGdNEWdn0
Принципы:
| $ scrapy runspider txspider.py | |
| 2016-07-05 23:11:39 [scrapy] INFO: Scrapy 1.1.0 started (bot: scrapybot) | |
| 2016-07-05 23:11:39 [scrapy] INFO: Overridden settings: {} | |
| 2016-07-05 23:11:40 [scrapy] INFO: Enabled extensions: | |
| ['scrapy.extensions.corestats.CoreStats', 'scrapy.extensions.logstats.LogStats'] | |
| 2016-07-05 23:11:40 [scrapy] INFO: Enabled downloader middlewares: | |
| ['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware', | |
| 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware', | |
| 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware', | |
| 'scrapy.downloadermiddlewares.retry.RetryMiddleware', |
Just a quickie test in Python 3 (using Requests) to see if Google Cloud Vision can be used to effectively OCR a scanned data table and preserve its structure, in the way that products such as ABBYY FineReader can OCR an image and provide Excel-ready output.
The short answer: No. While Cloud Vision provides bounding polygon coordinates in its output, it doesn't provide it at the word or region level, which would be needed to then calculate the data delimiters.
On the other hand, the OCR quality is pretty good, if you just need to identify text anywhere in an image, without regards to its physical coordinates. I've included two examples:
####### 1. A low-resolution photo of road signs
wget http://s3.amazonaws.com/alexa-static/top-1m.csv.zip
7x x top-1m.csv.zip
(Task for GCI 2015-16) Year: 2015-16 This guide will show how to create a simple login page with Flask (a python microframework) and a sqlite3 database.
As we are creating a simple user login app we need to store 2 basic values in the database: the username and the password. In order to build the database we need to define a schema:
| [Unit] | |
| Description=Python Virtualenv service example | |
| After=network.target | |
| [Service] | |
| User=my_user | |
| Group=my_group | |
| Environment=VIRTUAL_ENV=/home/my_user/.virtualenvs/my_venv | |
| Environment=PATH=$VIRTUAL_ENV/bin:$PATH | |
| ExecStart=$VIRTUAL_ENV/bin/my_entrypoint |
Normally the installation of MySQL can be achieved with a single command, which executes a script provided by MacMiniVault :
bash <(curl -Ls http://git.io/eUx7rg)
However, at the time of writing the script is not compatible with OS X El Capitan (10.11)
An alternative to the aforementioned installation script is installing MySQL using Homebrew. This gist assumes you already have Homebrew installed, if not first read the article "Homebrew and El Capitan"
Make sure Homebrew has the latest formulae, so run brew update first
Install phantomjs (http://phantomjs.org)
brew has a recipe for phantomjs, but it has moved on to version 2.0.0, and we still require 1.8.2 (at newest), and we can't get that old and dirty with brew anymore.
On a Mac:
mkdir -p ~/root
wget https://phantomjs.googlecode.com/files/phantomjs-1.8.2-macosx.zip -O ~/root/phantomjs-1.8.2-macosx.zip
| #!/bin/bash | |
| # | |
| # scrapyd This shell script enables scrapyd server on boot | |
| # | |
| # Author: Aaron Silber <aaron@brandthropology.com> | |
| # | |
| # chkconfig: - 50 01 | |
| # | |
| # description: Autostart scrapyd web scraper framework daemon | |
| # processname: scrapyd |