Skip to content

Instantly share code, notes, and snippets.

@avleen
Created May 5, 2015 12:14
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save avleen/61aa7f349bd070bd2e88 to your computer and use it in GitHub Desktop.
Save avleen/61aa7f349bd070bd2e88 to your computer and use it in GitHub Desktop.
--- date.rb.orig 2015-05-05 10:38:49.000000000 +0000
+++ date.rb 2015-05-05 09:55:45.000000000 +0000
@@ -87,6 +87,15 @@
# default to updating the @timestamp field of the event.
config :target, :validate => :string, :default => "@timestamp"
+ # A date range within which we accept timestamps to be legitimate.
+ # If we get a timestamp outside +/- this many days, it gets written to the
+ # trash index.
+ config :daterange, :validate => :number, :default => 1
+
+ # A trash index, to which we write log lines that fall outside the allowed
+ # date range.
+ config :trashindex, :validate => :string, :default => "logstash-trash"
+
# LOGSTASH-34
DATEPATTERNS = %w{ y d H m s S }
@@ -118,6 +127,9 @@
# TODO(sissel): Need a way of capturing regexp configs better.
locale = parseLocale(@config["locale"][0]) if @config["locale"] != nil and @config["locale"][0] != nil
setupMatcher(@config["match"].shift, locale, @config["match"] )
+
+ # @daterange is meant to be in days, but Time.now does math in seconds.
+ @daterange = @daterange * 86400
end
def setupMatcher(field, locale, value)
@@ -207,6 +219,14 @@
event[@target] = Time.at(epochmillis / 1000, (epochmillis % 1000) * 1000).utc
#event[@target] = Time.at(epochmillis / 1000.0).utc
+ # Make sure this date is inside the acceptable time range
+ now = Time.now
+ if !(now-@daterange..now+@daterange).cover?(event[@target])
+ @logger.warn("Received an event with a timestamp outside the approved range",
+ :text => event["message"])
+ event["esindex"] = @trashindex
+ end
+
@logger.debug? && @logger.debug("Date parsing done", :value => value, :timestamp => event[@target])
filter_matched(event)
rescue StandardError, JavaException => e
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment