Skip to content

Instantly share code, notes, and snippets.

View wlmcewen's full-sized avatar

Wade McEwen wlmcewen

16 followers · 9 following
View GitHub Profile
@wlmcewen
wlmcewen / sidekiq.log
Created May 10, 2017 14:42
CUR-20369
[flexadm@worker6.flex current]$ grep e597f8d3e0ff8086deac9b7a log/sidekiq.log
2017-05-08T22:33:18.952Z 1501 TID-oven4udh4 SingleListingEmailWorker JID-e597f8d3e0ff8086deac9b7a INFO: start
2017-05-08T22:33:18.990Z 1501 TID-oven4udh4 SingleListingEmailWorker JID-e597f8d3e0ff8086deac9b7a INFO: Prepairing email from=Dustin@RealtorDustin.com to=scottdreed18@yahoo.com
2017-05-08T22:33:19.284Z 1501 TID-oven4udh4 SingleListingEmailWorker JID-e597f8d3e0ff8086deac9b7a ERROR: Failure in SingleListingEmailWorker caused by ActionView::Template::Error with message undefined method `each' for nil:NilClass
2017-05-08T22:33:19.331Z 1501 TID-oven4udh4 SingleListingEmailWorker JID-e597f8d3e0ff8086deac9b7a ERROR: /home/httpd/gabby_gale/prod/releases/20170505151128/lib/standard_business_card.rb:112:in `use_marketing_name?'
2017-05-08T22:33:19.437Z 1501 TID-oven4udh4 SingleListingEmailWorker JID-e597f8d3e0ff8086deac9b7a INFO: [ProspectingTrace]: prospector_id=7baf3d6b025dd4980eddfed715e17e99 listing_key=20170508170032820501000000
@wlmcewen
wlmcewen / debug.sql
Created February 22, 2017 16:40
ARMLS-2782
prospector=# select source_id, event, event_created_at, newsfeed_id from newsfeed_item_event where newsfeed_item_id = 1229312741
order by event_created_at desc;
source_id | event | event_created_at | newsfeed_id
----------------------------+-------+----------------------------+-------------
20160910213418695892000000 | 4 | 2017-02-15 22:02:41.580145 | 12693041
20160910213418695892000000 | 5 | 2017-01-28 11:46:03.819968 | 12693041
20160910213418695892000000 | 1 | 2016-12-05 11:16:32.197197 | 12693041
20160910213418695892000000 | 0 | 2016-09-10 19:29:29.05982 | 12693041
prospector=# select current_status, resource_match_status, newsfeed_item_updated_at, newsfeed_item_created_at from newsfeed_item where newsfeed_id = 12693041 and source_id = '20160910213418695892000000';
@wlmcewen
wlmcewen / spark.rb
Last active September 30, 2016 16:33
API-3606
SparkApi:048:0* post("/savedsearches", "Name" => "Wade's bagley test", "Filter" => "City Eq 'Bagley' And MlsStatus Eq 'Active' And PropertyType Eq 'A'")
[{"ResourceUri"=>"/v1/savedsearches/20160930110443923409000000", "Id"=>"20160930110443923409000000", "OwnerId"=>"20081211210137031788000000", "Name"=>"Wade's bagley test", "Description"=>nil, "ContactIds"=>[], "UnsubscribedContactIds"=>[], "ModificationTimestamp"=>"2016-09-30T11:04:43-05:00", "Filter"=>"City Eq 'Bagley' And MlsStatus Eq 'Active' And PropertyType Eq 'A'", "QuickSearchId"=>nil, "Tags"=>[], "Provided"=>false}]
SparkApi:049:0> get("/newsfeeds/20151117150944963050973798")[{"Id"=>"20151117150944963050973798", "ResourceUri"=>"/v1/newsfeeds/20151117150944963050973798", "OwnerId"=>"20081211210137031788000000", "ModificationTimestamp"=>"2016-04-13T18:15:52Z", "CreatedTimestamp"=>"2015-11-17T21:09:44Z", "Events"=>[nil], "UseDefaultEvents"=>false, "Type"=>"SavedSearch", "Name"=>"alpha agent scheduled OpenHouse", "Active"=>true, "NotificationsActive"=>
@wlmcewen
wlmcewen / 2013-01-07-profiling-memory-leaky-sidekiq-applications-with-ruby-2.1.md
Created August 23, 2016 04:29 — forked from krasnoukhov/2013-01-07-profiling-memory-leaky-sidekiq-applications-with-ruby-2.1.md
Profiling memory leaky Sidekiq applications with Ruby 2.1

My largest Sidekiq application had a memory leak and I was able to find and fix it in just few hours spent on analyzing Ruby's heap. In this post I'll show my profiling setup.

As you might know Ruby 2.1 introduced a few great changes to ObjectSpace, so now it's much easier to find a line of code that is allocating too many objects. Here is great post explaining how it's working.

I was too lazy to set up some seeding and run it locally, so I checked that test suite passes when profiling is enabled and pushed debugging to production. Production environment also suited me better since my jobs data can't be fully random generated.

So, in order to profile your worker, add this to your Sidekiq configuration:

if ENV["PROFILE"]
@wlmcewen
wlmcewen / demo.rb
Created August 12, 2016 20:30
net_json
require 'uri'
require 'net/http'
require 'net_json'
class N
include NetJson
end
uri = URI.parse("http://internal.flexmls.com/flexiapi/prefs/auto_inbox_exstatus")
@wlmcewen
wlmcewen / customfields_ne.sql
Last active March 1, 2016 17:06
SELECT
"chs_coresearch"."listingkey",
"chs_coresearch"."modificationtimestamp",
"chs_coresearch"."sparkmodificationtimestamp"
FROM "chs_coresearch"
WHERE
(
"chs_coresearch"."listingkey" in
(
select distinct "chs_coresearch"."listingkey" from chs_coresearch
@wlmcewen
wlmcewen / Quieting some workers and firing up newsfeed workers
Created January 9, 2016 17:24
managing processes w/o the init scripts
# Note the uniq pid files at the end
/usr/bin/env bundle exec sidekiqctl quiet /home/httpd/prospector/prod/current/tmp/sidekiq.pid-match-3
/usr/bin/env bundle exec sidekiqctl quiet /home/httpd/prospector/prod/current/tmp/sidekiq.pid-match-4
/usr/bin/env bundle exec sidekiqctl quiet /home/httpd/prospector/prod/current/tmp/sidekiq.pid-match-5
/usr/bin/env bundle exec sidekiq -d -i 0 -e production -C /home/httpd/prospector/prod/current/config/sidekiq.yml -L /home/httpd/prospector/prod/current/log/sidekiq.log -g prospector-prod -c 10 -q newsfeeds,4 -q processing,1 -P /home/httpd/prospector/prod/current/tmp/sidekiq.pid-0
/usr/bin/env bundle exec sidekiq -d -i 0 -e production -C /home/httpd/prospector/prod/current/config/sidekiq.yml -L /home/httpd/prospector/prod/current/log/sidekiq.log -g prospector-prod -c 10 -q newsfeeds,4 -q processing,1 -P /home/httpd/prospector/prod/current/tmp/sidekiq.pid-1
/usr/bin/env bundle exec sidekiq -d -i 0 -e production -C /home/httpd/prospector/prod/current/config/sidekiq.yml -L /
@wlmcewen
wlmcewen / query_1.sql
Created December 28, 2015 16:33
API-3685 testing 12/28
--- old
SELECT newsfeed_item_id FROM "newsfeed_item" WHERE (newsfeed_item_id IN (select distinct on (source_id) newsfeed_item_id from newsfeed_item where newsfeed_item.subscriber_id='20150623231104072663000000' AND ("notification_sent" = 't' AND "newsfeed_item_approved" = 't') order by source_id, (select event_created_at from newsfeed_item_event where newsfeed_item.newsfeed_item_id = newsfeed_item_event.newsfeed_item_id order by event_created_at desc limit 1) desc nulls last, last_event_at desc nulls last)) ORDER BY (select event_created_at from newsfeed_item_event where newsfeed_item.newsfeed_item_id = newsfeed_item_event.newsfeed_item_id order by event_created_at desc limit 1) desc nulls last, newsfeed_item.last_event_at desc nulls last, newsfeed_item.newsfeed_item_id desc LIMIT 200 OFFSET 0
---new
SELECT newsfeed_item.source_id, MIN(newsfeed_item.newsfeed_item_id) as newsfeed_item_id, MAX(COALESCE(event_created_at, newsfeed_item_created_at)) as last_event_at FROM "newsfeed_item" LEFT OUTER JOIN newsfe
@wlmcewen
wlmcewen / old_crap.sql
Last active December 28, 2015 15:51
API-3685
--- QUERY1 : Prospector newsfeed scheduled email filter
SELECT source_id, newsfeed_item_id FROM "newsfeed_item" WHERE "newsfeed_item"."newsfeed_id" = 9390420 AND (0 < (select count(*) from newsfeed_item_event e where e.newsfeed_item_id = newsfeed_item.newsfeed_item_id and event in (1,7,4,2,0,5) and e.event_created_at >= '2015-12-15 19:39:34.940333')) ORDER BY (select event_created_at from newsfeed_item_event where newsfeed_item.newsfeed_item_id = newsfeed_item_event.newsfeed_item_id order by event_created_at desc limit 1) desc nulls last, newsfeed_item.last_event_at desc nulls last, newsfeed_item.newsfeed_item_id desc LIMIT 50;
--- QUERY2 : Ordering... subscribers?
SELECT subscriber_id, max ( GREATEST ( last_event_at, event_created_at ) ) AS last_event_at FROM "newsfeed_item" LEFT OUTER JOIN "newsfeed_item_event" ON "newsfeed_item_event"."newsfeed_item_id" = "newsfeed_item"."newsfeed_item_id" WHERE "newsfeed_item"."source_type" = '' AND "newsfeed_item"."subscriber_id" IN ( ... ) AND "newsfeed_item"."notif
@wlmcewen
wlmcewen / gist:dce5c9b20dfd3822ceb6
Created December 15, 2015 23:09
API-3622 brainstorming settings config for prospector.
# maybe we can handle the cascade db side like so.
select idx(array[22740185, 19161654], notification_id), * from notification where notification_id in (19161654, 52740185) order by idx(array[52740185, 19161654], notification_id) nulls last;
# perhaps we cascade by throwing the contact, agent and mls in a subselect on this collection ordered smartly
prospector_alpha=# select subscriber_id, master_owner_id, r.mls_id from resource r inner join newsfeed f on r.resource_id = f.resource_id where r.resource_id = '20151215223132896052000000' and f.subscription_name in (select subscription_name from newsfeed f2 where subscriber_id in (f.subscriber_id,r.master_owner_id, f.mls_id) and subscription_name = 'Subscription kaplan search ash' limit 1);
subscriber_id | master_owner_id | mls_id
----------------------------+----------------------------+----------------------------
20150320225609690019000000 | 20150320225609690019000000 | 20141203195816310156000000