electrical

logstash queries graphed with graphite.

Operation: Decouple whisper from graphite.

Method: Create a graphite function that does a date histogram facet query against elasticsearch for a given query string for the time period viewed in the current graph.

Reason: graphite has some awesome math functions. Wouldn't it be cool if we could use those on logstash results?

The screenshot below is using logstash to watch the twitter stream of keywords "iphone" "apple" and "samsung" - then I graph them each, so we get an idea of popularity. As a bonus, I also do a movingAverage() on the iphone curve to show you why this is awesome.

electrical

filter {

  grok {
    patterns_dir => '/root/ls-patterns'
    pattern => [ "%{STUNNELSERVICE}", "%{STUNNELCONNECT}", "%{STUNNELGENERAL}" ]
  }

}

electrical

  def searchin(array, fieldname, searchfor)
    if h = array.find { |h|
      if h[fieldname].is_a(Array)
        l = h[fieldname].find { |l| l[fieldname] == searchfor }
      else
        h[fieldname] == searchfor
      end
      }
      return h
    else

electrical

centos-65-x64 $  mktemp -t apply_manifest.pp.XXXXXX  
/tmp/apply_manifest.pp.13iMMS

centos-65-x64 executed in 0.04 seconds
localhost $ scp /tmp/beaker20140205-13063-zp0fg3 centos-65-x64:/tmp/apply_manifest.pp.13iMMS {}

centos-65-x64 $ env PATH="/usr/bin:/opt/puppet-git-repos/hiera/bin:${PATH}" RUBYLIB="/opt/puppet-git-repos/hiera/lib:/opt/puppet-git-repos/hiera-puppet/lib:${RUBYLIB}" puppet apply  --verbose --detailed-exitcodes /tmp/apply_manifest.pp.13iMMS
info: Loading facts in /etc/puppet/modules/stdlib/lib/facter/root_home.rb
info: Loading facts in /etc/puppet/modules/stdlib/lib/facter/puppet_vardir.rb
info: Loading facts in /etc/puppet/modules/stdlib/lib/facter/facter_dot_d.rb

electrical

[centos-65-x64] Waiting for machine to boot. This may take a few minutes...
The guest machine entered an invalid state while waiting for it
to boot. Valid states are 'starting, running'. The machine is in the
'aborted' state. Please verify everything is configured
properly and try again.

If the provider you're using has a GUI that comes with it,
it is often helpful to open that and watch the machine, since the
GUI often has more helpful error messages than Vagrant can retrieve.
For example, if you're using VirtualBox, run `vagrant up` while the

electrical

$ git diff rakelib/artifacts.rake
diff --git a/rakelib/artifacts.rake b/rakelib/artifacts.rake
index 5cf5321..9ee527e 100644
--- a/rakelib/artifacts.rake
+++ b/rakelib/artifacts.rake
@@ -10,18 +10,14 @@ namespace "artifact" do
     "CONTRIBUTORS",
     "{bin,lib,spec,locales}/{,**/*}",
     "patterns/**/*",
-    "vendor/elasticsearch/**/*",

electrical

require 'logstash-output-elasticsearch-ec2_jars.rb'

module LogStash::Outputs::ElasticSearch::Ec2

  def self.included(base)
    base.extend(self)
    base.register
  end

  def register

electrical

[2015-07-02 15:56:32,977][DEBUG][bootstrap                ] java.class.path: :/usr/share/elasticsearch/lib/elasticsearch-2.0.0-SNAPSHOT.jar:/usr/share/elasticsearch/lib/antlr-runtime-3.5.jar:/usr/share/elasticsearch/lib/lucene-memory-5.2.1.jar:/usr/share/elasticsearch/lib/lucene-spatial-5.2.1.jar:/usr/share/elasticsearch/lib/asm-4.1.jar:/usr/share/elasticsearch/lib/lucene-suggest-5.2.1.jar:/usr/share/elasticsearch/lib/joda-convert-1.2.jar:/usr/share/elasticsearch/lib/lucene-highlighter-5.2.1.jar:/usr/share/elasticsearch/lib/lucene-queryparser-5.2.1.jar:/usr/share/elasticsearch/lib/commons-cli-1.2.jar:/usr/share/elasticsearch/lib/joda-time-2.8.jar:/usr/share/elasticsearch/lib/jts-1.13.jar:/usr/share/elasticsearch/lib/lucene-analyzers-common-5.2.1.jar:/usr/share/elasticsearch/lib/jna-4.1.0.jar:/usr/share/elasticsearch/lib/jsr166e-1.1.0.jar:/usr/share/elasticsearch/lib/t-digest-3.0.jar:/usr/share/elasticsearch/lib/compress-lzf-1.0.2.jar:/usr/share/elasticsearch/lib/asm-commons-4.1.jar:/usr/share/elasticsearch/li

electrical

15:23:24.334 [main] DEBUG i.n.u.i.l.InternalLoggerFactory - Using SLF4J as the default logging framework
15:23:24.343 [main] DEBUG i.n.util.internal.PlatformDependent0 - java.nio.Buffer.address: available
15:23:24.344 [main] DEBUG i.n.util.internal.PlatformDependent0 - sun.misc.Unsafe.theUnsafe: available
15:23:24.345 [main] DEBUG i.n.util.internal.PlatformDependent0 - sun.misc.Unsafe.copyMemory: available
15:23:24.346 [main] DEBUG i.n.util.internal.PlatformDependent0 - java.nio.Bits.unaligned: true
15:23:24.410 [main] DEBUG i.n.util.internal.PlatformDependent - UID: 1000
15:23:24.410 [main] DEBUG i.n.util.internal.PlatformDependent - Java version: 8
15:23:24.410 [main] DEBUG i.n.util.internal.PlatformDependent - -Dio.netty.noUnsafe: false
15:23:24.410 [main] DEBUG i.n.util.internal.PlatformDependent - sun.misc.Unsafe: available
15:23:24.411 [main] DEBUG i.n.util.internal.PlatformDependent - -Dio.netty.noJavassist: false

electrical

12:23:24.968 [main] DEBUG i.n.u.i.l.InternalLoggerFactory - Using SLF4J as the default logging framework
12:23:24.975 [main] DEBUG i.n.util.internal.PlatformDependent0 - java.nio.Buffer.address: available
12:23:24.977 [main] DEBUG i.n.util.internal.PlatformDependent0 - sun.misc.Unsafe.theUnsafe: available
12:23:24.978 [main] DEBUG i.n.util.internal.PlatformDependent0 - sun.misc.Unsafe.copyMemory: available
12:23:24.979 [main] DEBUG i.n.util.internal.PlatformDependent0 - java.nio.Bits.unaligned: true
12:23:25.045 [main] DEBUG i.n.util.internal.PlatformDependent - UID: 1000
12:23:25.046 [main] DEBUG i.n.util.internal.PlatformDependent - Java version: 8
12:23:25.046 [main] DEBUG i.n.util.internal.PlatformDependent - -Dio.netty.noUnsafe: false
12:23:25.046 [main] DEBUG i.n.util.internal.PlatformDependent - sun.misc.Unsafe: available
12:23:25.046 [main] DEBUG i.n.util.internal.PlatformDependent - -Dio.netty.noJavassist: false