Skip to content

Instantly share code, notes, and snippets.

@rashidkpc
Last active December 15, 2015 19:39
Show Gist options
  • Save rashidkpc/5312870 to your computer and use it in GitHub Desktop.
Save rashidkpc/5312870 to your computer and use it in GitHub Desktop.
JSON to timestamped index importer
require 'json'
require 'time'
require 'tire'
# File to read
file = 'test.txt'
# Field in which time value can be found
timefield = '@timestamp'
# Index and date pattern to use
index_pattern = 'myindex-%Y.%m.%d'
events = {}
IO.foreach(file) do |line|
event = JSON.parse(line) # You might be able to use split or something to get attributes
time = Time.parse(event[timefield])
index = time.strftime(index_pattern)
if !events.has_key?(index)
events[index] = []
end
events[index] << event
end
events.each do |index, logs|
puts "Importing: #{logs.size} into #{index}"
Tire.index index do
import logs
end
end
{"id":7059,"type":"apache","@timestamp":"2013-04-03T08:46:27Z","ip":"145.37.131.46","request":"/HKEK.css","bytes":7168}
{"id":7060,"type":"apache","@timestamp":"2013-04-03T15:34:53Z","ip":"157.127.34.13","request":"/HHEH.html","bytes":6978}
{"id":7061,"type":"apache","@timestamp":"2013-04-06T16:42:02Z","ip":"61.240.84.142","request":"/EHKK.php","bytes":8898}
{"id":7062,"type":"apache","@timestamp":"2013-04-05T15:05:12Z","ip":"152.106.197.128","request":"/EEEE.png","bytes":5588}
{"id":7063,"type":"apache","@timestamp":"2013-04-02T08:44:23Z","ip":"39.189.137.196","request":"/HKKE.png","bytes":6386}
{"id":7064,"type":"apache","@timestamp":"2013-03-31T10:11:16Z","ip":"178.211.189.140","request":"/EEKH.php","bytes":9739}
{"id":7065,"type":"apache","@timestamp":"2013-04-02T13:31:17Z","ip":"177.5.17.106","request":"/EEKE.png","bytes":2440}
{"id":7066,"type":"apache","@timestamp":"2013-04-01T22:10:17Z","ip":"83.59.80.235","request":"/HEEK.png","bytes":3399}
{"id":7067,"type":"apache","@timestamp":"2013-04-04T08:53:40Z","ip":"11.240.130.47","request":"/HHEE.html","bytes":6572}
{"id":7068,"type":"apache","@timestamp":"2013-04-02T16:13:43Z","ip":"19.212.95.116","request":"/KEEK.gif","bytes":2448}
{"id":7069,"type":"apache","@timestamp":"2013-04-03T11:31:30Z","ip":"137.61.97.2","request":"/HEEH.html","bytes":0}
{"id":7070,"type":"apache","@timestamp":"2013-04-02T05:35:44Z","ip":"129.184.47.164","request":"/HEEE.html","bytes":0}
{"id":7071,"type":"apache","@timestamp":"2013-03-31T13:57:02Z","ip":"208.23.124.245","request":"/HKHH.php","bytes":3844}
{"id":7072,"type":"apache","@timestamp":"2013-04-03T22:19:06Z","ip":"32.95.204.207","request":"/EHHK.html","bytes":3666}
{"id":7073,"type":"apache","@timestamp":"2013-04-02T02:48:50Z","ip":"203.170.229.190","request":"/EHHK.php","bytes":2649}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment