avleen

package main

import (
        "bufio"
        "fmt"
        "github.com/saintienn/go-spamc"
        "io/ioutil"
        "log"
        "os"
)

avleen

### Keybase proof

I hereby claim:

  * I am avleen on github.
  * I am avleen (https://keybase.io/avleen) on keybase.
  * I have a public key whose fingerprint is 1ABD 5DE1 7C47 21B5 4B71  AEC7 F262 CCCC 12B5 B5BD

To claim this, I am signing this object:

avleen

{
    "template": "logstash-*",
    "settings" : {
        "number_of_shards" : 180,
        "number_of_replicas" : 1,
        "index.refresh_interval" : "5s",
        "index.routing.allocation.total_shards_per_node" : 4,
        "index.search.slowlog.threshold.query.warn": "100ms",
        "index.search.slowlog.threshold.fetch.warn": "100ms",

avleen

A multi-threaded process which listens on port 80 for connections from yum, apt, npm, gem, etc.
It recognises the kind of request made (yum, apt, npm, gem) automatically and responds accordingly.

If a requested package is available locally, it is served.
If not, it is fetched from a configured list of upstream servers, cached and served.

On a regular basis, the process retreives a package list (eg, Repodata.xml) from its upstream servers,
and merges it with a list of local packages.

Users provide mappings for where to point for each repo (eg: cent 7, cent 7 updates, npm, etc). Eg:

avleen

--- date.rb.orig	2015-05-05 10:38:49.000000000 +0000
+++ date.rb	2015-05-05 09:55:45.000000000 +0000
@@ -87,6 +87,15 @@
   # default to updating the @timestamp field of the event.
   config :target, :validate => :string, :default => "@timestamp"
 
+  # A date range within which we accept timestamps to be legitimate.
+  # If we get a timestamp outside +/- this many days, it gets written to the
+  # trash index.
+  config :daterange, :validate => :number, :default => 1

avleen

            "dynamic_templates": [
                {
                    "long_template" : {
                        "match" : "l_*",
                        "mapping" : { "type" : "long", "index" : "not_analyzed" }
                    }
                },
                {
                    "double_template" : {
                        "match" : "d_*",

avleen

{
    "template": "logstash-*",
    "settings" : {
        "number_of_shards" : 24,
        "number_of_replicas" : 1,
        "index.refresh_interval" : "5s",
        "index.routing.allocation.total_shards_per_node" : 1,
        "index.search.slowlog.threshold.query.warn": "10s",
        "index.query.default_field" : "message",
        "analysis" : {

avleen

"<generator" daemon prio=10 tid=0x00007f4bc0088000 nid=0x72f4 runnable [0x00007f4c5435c000]
   java.lang.Thread.State: RUNNABLE
        at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
        at org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
        at org.jruby.evaluator.ASTInterpreter.INTERPRET_BLOCK(ASTInterpreter.java:112)
        at org.jruby.runtime.Interpreted19Block.evalBlockBody(Interpreted19Block.java:206)
        at org.jruby.runtime.Interpreted19Block.yield(Interpreted19Block.java:194)
        at org.jruby.runtime.Interpreted19Block.call(Interpreted19Block.java:125)
        at org.jruby.runtime.Block.call(Block.java:101)
        at org.jruby.RubyProc.call(RubyProc.java:290)

avleen

• Small gathering at Chez McDonnell, to celebrate how far we've come in the last year.
• The project has taken on a life of its own.
• Regularly getting mentioned on twitter, in a completely organic way.
• Found two examples (Ubuntu server and ATS mailing lists) where people are using OpsSchool as examples on how to do documentation well! Where did this come from?!
• Some stats, from Google Analytics for ~Q4, compared to ~Q3:
• Average visit duration is up (3mins, vs 2mins 18secs).
• Pages per visit is up (3, vs 2.5).
• Unique visitors is down (7.3k vs 11.2k) but this is OK! We had a huge spike in July which threw this off. Otherwise this is more comparable.
• A lot of traffic is "direct" (probably things like the Devops newsletter, etc), followed closely by Google, then twitter, reddit and my blog post on "So y ou want to be a systems administrator?" (which should have read "operations engineer", but the old title is familiar to people).
• Popular pages are the introduction followed by "sysadmin 101"

avleen

{ "comment": "
## We have a lot of data (~2Tb/day), and ~12 ES nodes, so we set the number of
## shards to 10, to distribute the logs roughly evenly. If you have less data,
## use fewer shards.
## It doesn't really make sense to have more shards, than you have servers.
##
## The index.routing.allocation.require.tag is used internally to make new
## indices on fast hardware. We run a nightly cron which moves older indices
## to slower hardware: https://gist.github.com/avleen/ecfb48f49d260b25e45e
##