Skip to content

Instantly share code, notes, and snippets.

View IanVaughan's full-sized avatar
👯‍♀️
OMG

Ian Vaughan IanVaughan

👯‍♀️
OMG
50 followers · 121 following
View GitHub Profile
@IanVaughan
IanVaughan / Heroku grok logstash rule for Heroku Deployments.md
Last active January 20, 2021 07:20
Heroku grok logstash rule for Heroku Deployments

Sample:

Deploy 77bd8aca by user some@email.com

Rule:

rule1 Deploy %{notSpace:deploy.sha} by user %{data:deploy.user}
@IanVaughan
IanVaughan / Datadog grok logstash rules for standard sidekiq logs.md
Last active January 20, 2021 07:03
Datadog grok logstash rules for standard sidekiq logs

Sample:

I, [2020-04-05T20:10:03.013349 #4]  INFO -- : [Paypal::Transactions] jid:39d5996400fa48a, too account_id:3, end_date:2020-04-04T23:59:59Z, page:1

Rules:

sidekiq %{notSpace:log.letter},\s+\[%{date("yyyy-MM-dd'T'HH:mm:ss.SSSSSS"):log.date}\s+#.\]\s+%{word:log.level}\s+--\s+\:\s+\[%{notSpace:sidekiq.job.class}]\s+jid:%{word:sidekiq.job.jid}%{data::keyvalue(":", "", ",")}
@IanVaughan
IanVaughan / Datadog grok logstash rules for custom sidekiq logs.md
Last active January 20, 2021 07:03
Datadog grok logstash rules for custom sidekiq logs

Samples:

I, [2020-11-12T12:12:18.883446 #4]  INFO -- : sidekiq-stats {"processed":48469869,"failed":30091,"scheduled_size":0,"retry_size":0,"dead_size":0,"processes_size":1,"default_queue_latency":0,"workers_size":0,"enqueued":0}
I, [2020-11-12T12:03:11.112090 #4]  INFO -- : sidekiq-queues [[{"name":"default"},{"count":0},{"size":0},{"latency":0}],[{"name":"import"},{"count":0},{"size":0},{"latency":0}],[{"name":"low"},{"count":26},{"size":26},{"latency":183.94694018363953}]]

Rules:

autoFilledRule1 I,\s+\[%{date("yyyy-MM-dd'T'HH:mm:ss.SSSSSS"):date}\s+#4\]\s+%{word:level}\s+--\s+\:\s+sidekiq-stats\s+\{\"processed\"\:%{integer:processed},\"failed\"\:%{integer:failed},\"scheduled_size\"\:%{integer:scheduled_size},\"retry_size\"\:%{integer:retry_size},\"dead_size\"\:%{integer:dead_size},\"processes_size\"\:%{integer:processes_size},\"default_queue_latency\"\:%{number:default_queue_latency},\"workers_size\"\:%{integer:workers_size},\"enqueued\"\:%{integer:enqueued}\}
autoFilledRule2 I,\s+\[%{date("yy
@IanVaughan
IanVaughan / gist:b604d5b26a0b8fa36c4fab9188cdb6db
Created September 18, 2020 11:15
failed to build gem native extension. xcode
Fetching nio4r 2.5.4
Installing nio4r 2.5.4 with native extensions
Gem::Ext::BuildError: ERROR: Failed to build gem native extension.
current directory: /Users/ianvaughan/.asdf/installs/ruby/2.7.1/lib/ruby/gems/2.7.0/gems/nio4r-2.5.4/ext/nio4r
/Users/ianvaughan/.asdf/installs/ruby/2.7.1/bin/ruby -I /Users/ianvaughan/.asdf/installs/ruby/2.7.1/lib/ruby/2.7.0 -r ./siteconf20200918-40223-cj3k1e.rb extconf.rb
checking for unistd.h... *** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of necessary
libraries and/or headers. Check the mkmf.log file for more details. You may
need configuration options.
@IanVaughan
IanVaughan / hash-to-json.rb
Created September 15, 2020 10:42
Convert a log of hashes into json files in ruby
i=0
file = File.read('log')
file.each_line do |line|
hash = eval line
File.write("data-#{i}.json", hash.to_json)
puts i
i+=1
end
@IanVaughan
IanVaughan / upgrade_db.sh
Created August 27, 2020 11:21
Steps to Upgrade a Heroku DB in a script
#!/usr/bin/env bash
# https://devcenter.heroku.com/articles/updating-heroku-postgres-databases#updating-with-follower-changeover
APP=name-production # name-staging
heroku addons:create heroku-postgresql:standard-2 --follow DATABASE_URL --app $APP # DONE
heroku addons:wait # DONE
for (( ; ; )); do; heroku pg:info --app $APP | grep Behind; sleep 1; done # DONE
@IanVaughan
IanVaughan / gist:9ac9a30597719b3e86d6b252e0db9c92
Created August 27, 2020 11:19
Intresting differences between rails routing form_for helpers
# [:admin, @application] => admin_onboarding_url
# [:admin, @collections_facility] => admin_collections_facility_url
# [:admin, @application, @collections_facility] => admin_onboarding_collections_facility_url
# [:admin, :application, @collections_facility] => No route matches {:action=>"show", :application_id=>#<Collections::Facility id: 409, user_id: 1303, amount: 0.1e5, revenue_share: 0.25e1, fee: 0.31e1, created_at: "2020-01-24 10:45:50", updated_at: "2020-01-24 10:45:50", application_id: 886>, :controller=>"admin/collections/facilities"}, missing required keys: [:id]
# [:admin, :application, @application] => admin_application_onboarding_url
# format.html { redirect_to [:admin, :application, @application], notice: 'Facility was successfully created.' }
# redirect_to [:admin, :application, @collections_facility]
# => "<html><body>You are being <a href=\"http://test.host/admin/applications/233/collections/facilities/233\">redirected</a>.</body></html>"
@IanVaughan
IanVaughan / team_stats.rb
Created August 27, 2020 11:17
Find GitHub team PR counts
# frozen_string_literal: true
# https://gist.github.com/borischerkasky/7acc8b19e7b5c6b044bf92ab6f467126
require 'awesome_print'
require 'httparty'
class TeamPrStatistics
attr_accessor :stats
@IanVaughan
IanVaughan / unused_routes.rb
Created August 27, 2020 11:15
Find unused rails routes
# frozen_string_literal: true
Rails.application.eager_load!
unused_routes = {}
# Iterating over all non-empty routes from RouteSet
Rails.application.routes.routes.map(&:requirements).reject(&:empty?).each do |route|
name = route[:controller].camelcase
next if name.start_with?('Rails')
@IanVaughan
IanVaughan / gist:806cd9cb6be5995b8ca9ef0be1856e1a
Created August 27, 2020 11:14
Mounting a raid drive on my Netgear readynas
mount
fsck /dev/sdb1
parted /dev/sda 'print'
lsblk -f
blkid
mount /dev/sba1 ~/mountpoint
mdadm --assemble --run /dev/md0 /dev/sdb1
mdadm --assemble --run /dev/md3 /dev/sdb3
mount /dev/md3 ~/old-disk/