liwh

// Based on tinrtgu's Python script here:
// https://www.kaggle.com/c/avazu-ctr-prediction/forums/t/10927/beat-the-benchmark-with-less-than-1mb-of-memory 
package main

import (
    "encoding/csv"
    "os"
    "strconv"
    "hash/fnv"
    "math"

liwh

If you want, I can try and help with pointers as to how to improve the indexing speed you get. Its quite easy to really increase it by using some simple guidelines, for example:

- Use create in the index API (assuming you can).
- Relax the real time aspect from 1 second to something a bit higher (index.engine.robin.refresh_interval).
- Increase the indexing buffer size (indices.memory.index_buffer_size), it defaults to the value 10% which is 10% of the heap.
- Increase the number of dirty operations that trigger automatic flush (so the translog won't get really big, even though its FS based) by setting index.translog.flush_threshold (defaults to 5000).
- Increase the memory allocated to elasticsearch node. By default its 1g.
- Start with a lower replica count (even 0), and then once the bulk loading is done, increate it to the value you want it to be using the update_settings API. This will improve things as possibly less shards will be allocated to each machine.
- Increase the number of machines you have so

liwh

[core]
# The home folder for airflow, default is ~/airflow
airflow_home = /Users/p1nox/airflow

# The folder where your airflow pipelines live, most likely a
# subfolder in a code repository
dags_folder = /Users/p1nox/airflow/dags

# The folder where airflow should store its log files. This location
base_log_folder = /Users/p1nox/airflow/logs

liwh

- certain endpoints are always blocked
if nginx_uri == "/_access_token" or nginx_uri == "/_me" then
    ngx.exit(403)
end

-- import requirements
local cjson = require "cjson"

-- setup some app-level vars
local app_id = "APP_ID"

liwh

Routes

小心地使用 Match（Rails 3 已实现）

Rails 3 提供了 match 方法供我们自定义 routes，然而我们要小心使用它以避免“跨站脚本攻击”（XSS Attack）。比如像这样的 routes：

注：（r3 代表 Rails 3，r4 代表 Rails 4）

# routes.rb

liwh

Fibur

Fibur is a library that allows concurrency during Ruby I/O operations without needing to make use of callback systems. Traditionally in Ruby, to achieve concurrency during blocking I/O operations, programmers would make use of Fibers and callbacks. Fibur eliminates the need for wrapping your I/O calls with Fibers and a callback. It allows you to write your blocking I/O calls the way you normally would, and still have concurrent execution during those I/O calls.

Example

Say you have a method that fetches data from a network resource:

liwh

❷ > QUEUE=* rake resque:work --trace
** Invoke resque:work (first_time)
** Invoke resque:preload (first_time)
** Invoke resque:setup (first_time)
** Execute resque:setup
** Execute resque:preload
rake aborted!
No such file to load -- devise/confirmations_controller
/Users/stefan/.rvm/gems/ruby-1.9.2-p290@my-rails-project/gems/activesupport-3.1.1/lib/active_support/dependencies.rb:306:in `rescue in depend_on'
/Users/stefan/.rvm/gems/ruby-1.9.2-p290@my-rails-project/gems/activesupport-3.1.1/lib/active_support/dependencies.rb:301:in `depend_on'

liwh

$ sequel postgres://mylogin:mypasswd@localhost/bddtest
Your database is stored in DB...
>> require 'memcache'
=> true
>> MemCache
=> MemCache
>> MemCache::VERSION
=> "1.5.0.1"
>> CACHE = MemCache.new("localhost")
=> MemCache: 1 servers, 1 buckets, ns: nil, ro: false

liwh

/* 
   Some simple Github-like styles, with syntax highlighting CSS via Pygments.
*/
body{
	font-family: helvetica, arial, freesans, clean, sans-serif;
	color: #333;
	background-color: #fff;
	border: none;
	line-height: 1.5;
	margin: 2em 3em;

liwh

Railties 3.1 Beta 1

• The -j option of the application generator accepts an arbitrary string. If passed "foo", the gem "foo-rails" is added to the Gemfile, and the application JavaScript manifest requires "foo" and "foo_ujs". As of this writing "prototype-rails" and "jquery-rails" exist and provide those files via the asset pipeline. Default is "jquery". [fxn]

• jQuery is no longer vendored, it is provided from now on by the jquery-rails gem. [fxn]

• Prototype and Scriptaculous are no longer vendored, they are provided from now on by the prototype-rails gem. [fxn]

• The scaffold controller will now produce SCSS file if Sass is available [Prem Sichanugrist]