https://www.youtube.com/watch?v=iCg6t5iedEw incorrect transcript

gistfile1.txt

https://www.youtube.com/watch?v=iCg6t5iedEw incorrect transcript

0:00 but tanker I'm inside I for one flows you that don't know me my name is no up
0:05 in countries meet up strapped I
0:07 about 4-5 months now and the first meet up I went to was actually
0:12 casper's sad rabbits ok and so
0:16 also ask one spot by asking to find an application for a while
0:20 and so we gave a guy since we could find a use case for it
0:23 and then you have no but foolish label so trust quite
0:27 ego I tweeted miss I
0:30 of the a reddish presentation which
0:34 van project s but can touch me and say hi would you mind coming home
0:37 and talking about y'all expenses with rabbit
0:41 which is why i cant for the first ok of the first meet up with 25th a non-state
0:46 here talking about not symphony
0:47 and the symphony mate up I'm so before we became
0:51 I has the slide about mayhem I am
0:54 so I'm a passionate developer thing isn't simply since about 2011
0:59 I run the Twitter product
1:02 for a major advertising I AP lab partner
1:06 and by in my spare time was spent time I have I'm spot studying physics at the
1:11 Open University
1:12 and I'm aight sweet about Sun selection development & Co says my apt 202 and
1:17 what their
1:18 if anybody wants to follow me so for so long that give you the
1:22 the background of our implementation why we needed to use rabbit and how I system
1:27 is involved
1:28 from the very first kind of curing system that we had to
1:32 to what we have now so first hole
1:36 I assay we are major social media I'd like consumer
1:40 we potentially make thousands API calls per hour
1:44 least thousands I P of course but I I'm
1:47 we made it to find a scalable solution because at current situation wasn't
1:52 working for us
1:53 at so us when you to find something that was more scalable
1:57 easier to expand than what we had previously allocating I threw that now
2:01 so the first generation
2:02 over our a violation was simply
2:06 we've having crime jobs running every minute to select X amount of rice from
2:10 the database to saint
2:11 so APIs I'll crimes are running simply task speech about his performance at me
2:17 one job
2:18 I'm we've got a sick person spawning multiple coal requests to local host
2:23 I'm which is a bit of a problem will get but which will get back to you in a
2:27 moment as well I'll explain why that's a problem for us
2:30 and ultimately what we're doing is sending requests back to the same server
2:34 to soap star of most vocal requests and that multiple
2:38 I got closer all at once I'm we've got you are simply tossed running
2:43 every minute I'm in fact
2:46 its 120 crumbs eight running every minute I'm
2:50 and a lot so that I'm not done actually do anything they're just makin bites by
2:54 schools on me
2:55 on the database instance and what turning into a result says there's no
2:59 real made for them axo answer I kinda redundant
3:03 which Showtime which we done one happening really
3:06 am and say we're just going to focus on the smoke of arab guy which is the
3:11 sinking
3:11 and yet creation that we've got going on and why we need skin is properly
3:16 so it was good example here of the first generation of curing system we've just
3:21 selection columns from that first table where the update flag is
3:25 you truck by problems with this approach is that we're relying on first we rely
3:30 on selecting
3:31 what we feel we can get away with within a minute
3:35 at which isn't very many it so what I'm say because we don't want to send the
3:38 same rose twice to the API and stop
3:41 incurring extra charges for our clients on
3:44 with Facebook I'm we might only be selection of euros but we need to select
3:49 from hundreds of thousands in the table
3:52 so of course so I borrow checking that's gonna be quite slow
3:55 if when only selecting 20 runs at a time and every minute we've got a massive
4:00 fighter
4:01 rates black on the database and only
4:04 web server CPA because the web server is running these contracts we don't have a
4:08 dedicated instance running
4:10 these at least awesome in it and
4:13 the tights comparison really is separate from done on the next concert we could
4:16 quite easily at
4:17 indexes to some these columns in space things up but at the moment we haven't
4:22 and the same process I told you about where we far-off
4:25 crumbs to so that were recalled of our of
4:28 API calls to our iron server actually ends up hope
4:32 bomb and saw about and it becomes unresponsive until we re boat it
4:36 which means they inhabit the air the clients that have anything to login so
4:40 and so and so if we voted
4:42 so the second attempt that we had
4:45 this the second generation set go it alone
4:50 we've just got a cute able where we enjoy net but somehow I'm again the same
4:55 so the kids I'll actually just retraction gross
4:58 waves the number of that elections are being cute
5:01 exs got a few benefits to it was selecting from
5:05 euros to start with so is a little bit faster when benchmark benchmark to it
5:09 but we do have indexes on this as well so is is a bit quicker
5:13 at problems with that is we can still only select a certain amount per minute
5:18 and we have to guess at what we think we can get away with the same for first
5:22 generation of the Carrie
5:23 and we still got a pic I to rate spike and
5:28 as well the complexity increases with this because I'm we end up with the
5:32 queue table
5:33 for for sending stuff the API happen we end up with a cute I will sometimes we
5:37 on a sense a price difference what you raiding and you need to select different
5:40 data from different tables
5:41 and you end up affecting different tables just to try and shoot
5:46 APR aids and rights and and his
5:49 a bunch mohair sigh we end up with Africa sample shown it to sink
5:54 tweaks from the AP Lang rushing progressed table
5:57 which is then join to the Promoted Tweets table
6:00 which is then joins where actual trade so these these assets selecting advert
6:04 traits from from Twitter acts I'm a target saying least join the sink
6:09 table to the targeting table and then to the actual table which
6:13 host the targeting and eggs it just becomes
6:16 a bit of a mess try try and do anything
6:19 when you are selecting from the API
6:23 on side I am the code as well becomes a little bit messy
6:28 and so we've got here on a campaign for instance
6:32 that we've got this key method which sets up
6:35 an instance of the 20 K to an agency which tank right here
6:40 inside let we've got
6:44 hear them efforts take much to share as well which sent where
6:47 have to posse and wall column we want to join on sales
6:52 its coupling the night spice to the code and little bit too much within this this
6:57 queue system and finally within hair
6:59 I N Save this is the the column that was just passed and we have to have mess
7:04 about with
7:05 detecting if the cure exists at by mother creating it we just return if it
7:10 doesn't exist only
7:11 make a new version of the cave in return back so everything's a little bit mess
7:15 saying it's not maintainable solution
7:17 science of the tasks as well what we've got
7:23 is speak and say here's some affection X campaigns from the database
7:27 which we've already been through and then inside that we're going to fetch
7:31 the access tokens from the tight space for each row
7:34 send it to the API and then say the response back to the right place again
7:38 we just got a helluva lot of noise so they have on the network
7:41 and control from the tight spice and
7:44 quick flow chart hey you can say the is the same saves our stuff
7:48 into the database might run out of picks up the information
7:52 looks through the rows free trial affection
7:55 the access token hey making a pure course ended up saving result
7:59 and just looking for him is a helluva lot more is going on there which one to
8:03 get rid of
8:03 and it finally we finish at we finish what seconds left so obviously this is
8:09 kinda slow this this is the reason that we needed to find something else today
8:13 so hence leahy but we
8:16 charged rabbit and we're just quickly go through what happens with rabbit
8:20 we have a producer which sends a message to read it so the rabbit will Bay
8:24 in your application code it could be in a doctrine her cry anything like that
8:28 and rabbit then saves a message queues message up
8:32 and the consumer does the actual out later on at you can say here the code is
8:37 no better idea hey
8:39 instead of having this Sat method in the nato to return 0
8:43 that QN stance we've now got I just kidding and then we use to make state AP
8:48 methods here so
8:49 crop crop is my point for Mac that we're just going to use this
8:52 it you can save this is just a little bit
8:57 nice I inside because now
9:00 we can just pass the access token straight into the message that we're
9:03 sending whether I get there there's no need now
9:06 track she selected from the database trace right that we've got
9:09 and then in here this is just sex not the rabbit connection
9:13 nice and simple in say the Q nine here
9:16 which which music different cues
9:20 means that you can perform different tasks really a sleigh
9:24 and keep everything nice and Russian separate the consumer can be written in
9:29 any language we
9:30 decided to have a look around see was that she was available
9:35 I'm choose something fast is that way
9:38 is what we came up with is my point spent on implementing rabbits then
9:41 choose a really slow consumer that's not gonna
9:44 actually give you any kinda benefits so what we thought about
9:48 was your life
9:51 I with fat with gonzo and eloquent I'll rm there
9:55 at we decided against that we've got some a
9:59 so weaver we've got we've got a lot of my service is being run with node.js
10:04 I should we decided to go with node
10:07 and see Christ sake allies pay and the
10:10 production equipment I'll RM finite and
10:14 side who that's why we went with my own
10:17 in Val not going to go into in tonight don't worry and so the benefits so this
10:23 is that we've now got a dedicated instance running night and
10:26 performing these tasks at Sir taken the the proper keep pressing
10:30 at queue processing even of the web server side the client doesn't see that
10:35 drop in performance every so often I might run a sense up to API
10:39 would reduce number fight Space Coast I'm where
10:43 continually process in case we don't have to worry about can we select
10:46 this amount and processing within a minute before the crime runs again
10:50 et cetera et cetera and we've got a much more simplified
10:54 architecture site necks is the main flow chart we've got
10:58 and say hey here's a size campaigns we send at the same time sunday from to buy
11:02 spice
11:03 we also send everything to rabbit
11:06 attention Iran's might shape you like o.o as a successful
11:10 yes save the result but that I twice we've got rid of if we compare
11:13 to the previous one we've got rid of everything else with the change but I'll
11:17 we go back to here as well she can say I think I
11:21 ocho wasn't successful we can
11:24 send it back to read it now that's one of the things that we had much
11:28 difficulty with was working out how to sense but that
11:30 how to reach you and whether it could read it doesn't have an actual wake you
11:34 method
11:34 at which surprise tusks we thought it would
11:38 and and there are many different
11:41 suggestions on the internet have to achieve it at but that brings us on Sat
11:46 return messages
11:47 so the way that we decided serve re Q messages when that when there was a
11:52 successful Co
11:53 was by using something called the dead letter exchange
11:56 which just basically maintenance but we've got cue i which I prime prom
12:01 weekend
12:01 the consumer post messages from
12:04 here make sure I P O I cope if it's not successful
12:07 we then put the message back into another kid corky Bay
12:12 in this example at Wave a couple of extra fields here so we've got
12:16 that lecture exchange points back to queue I
12:20 and we've got a tail on that message some nothing actually pose anything from
12:24 this k what so ever
12:25 and that Tice love 900,000 Mary suspect they minutes in
12:29 is in many second 6 15 minutes in
12:33 and that in that message said nothing actually can change from her so
12:37 and watch the teacher expires it goes back into cue I and then the consumer
12:42 picks up to make do the whole thing again and hopefully
12:44 neck so be successful without how hello how is it going from human
12:50 at rabbit just one city challenge lies read it takes the
12:54 if I go back to head the that better exchange with past and
12:58 rabbit will take and put it back in school nice to put it back in skewl a
13:03 where the when can she was going to pick up again size
13:07 at two points over what we've done is try try was removed thirteen different
13:12 crumbs
13:12 I'm so that's what about us
13:15 where continually processing the case we don't
13:19 we don't have to as a say we don't have to baby waiting a minute fat
13:23 everything so it's a process and then pick up another bunch everything happens
13:27 pretty much in real time
13:29 I'm would make the right away from clashing service which were just complex
13:33 it's a
13:33 and and everything is much faster and after school
13:38 graphs at the end to show you as well with that I
13:41 bad points because sweet charged
13:44 sequel ice we've had to remodel everything in sick wise
13:48 as well as in doctrine side but first or slowed us down learning how to use
13:53 equalize
13:54 sector vol yeah with we now want to maintain a mostly had a special %uh go
13:59 into places
14:00 which is a bit of a nightmare and we haven't yet decided how to tackle that
14:04 hmm
14:07 I'm so he's come across I did today with I was benchmarking our system with the
14:12 mayor system
14:13 and you can say hey that's at Wat
14:16 we've got one single large accounts organization Iran which took about six
14:20 hours
14:21 on the outs on the old method of doing is a huge account I
14:25 and it took 13 minutes I think to do it
14:28 you let so we've gone from over six hours 13 minutes
14:32 I'm and down here we had a couple with a
14:35 Cirque at about 45 minutes when I was like in half hour previously
14:40 sets rainy rainy suspect things out which means that clients gonna be
14:44 happier
14:45 as well and so essential time taken the next slide here
14:49 his actually them at a time that we site chance a the
14:53 the longer the original cue what i can.
14:57 the more time that way safe which I affects con self-explanatory rarely
15:02 anyway
15:03 reflects site what's next for us we're gonna implement rabbit cues for campaign
15:09 creation
15:09 on the Twitter site we haven't quite done that with 20 at
15:13 at we want to do it for Facebook
15:16 because it's the Facebook sync which is the one that way it for bombing
15:19 ourselves with such gonna be a major
15:21 Khan a plus point for us and a funny we've already bill I relax
15:25 implementation
15:26 directly on top a rather when bothered with anything else with bill the NIH
15:29 with rabbit
15:30 and that's working fantastic
15:33 and cider there we go thats may
15:37 once again fit for you guys that wanna from a and you might well tell
15:42 my age by now with that said the entrance
15:44 asylum other questions spat so
15:48 from my questions over even means that nobody's been listening north on a
15:52 really good presentation I think it's probably not the lecture
15:55 I when we have access to the mall in the
15:58 class make informer and consume yes so am
16:03 Brenda because we're saving by to back from the IP I say my baby
16:10 I'm so if were fetching site map tracks from the API
16:13 we need so what she say first race of Promoted Tweets which is a treatment to
16:18 an account
16:18 I'm and a metre and everything else with beading and budgeting everything
16:23 attached to a show that's what we need to safe expect if we could go away with
16:27 not remodeling it
16:28 that have been great and things would've been even faster for us to make a
16:31 weekend of so
16:32 launch that month earlier than we did but for Gino
16:36 God are you considering
16:39 things you have see allies and all remember at some point use
16:44 rabbit in families also
16:47 replace I whose angle wat a
16:52 always does home them send
16:55 down yes cautious so that would be I'm
16:59 opinion some bring some folks this life and that would
17:03 obviously baby preferable to any be maintained that in one place
17:07 and and perhaps the way to do it they may have everything to
17:11 tonight and I'm access via an API from from the past Bayside
17:16 I'm abuse issues profit modest you rolls eyes
17:19 a back all
17:22 actually results settings
17:26 us not yet my by my think about that yeah
17:29 purple to not help the data
17:35 notices really my Spanish PSP
17:38 because view no ready then you
17:41 be yet what I'm
17:45 for russia has been yeah and x-ray we've got a few legacy possible ap
17:49 time which were trying to drag up to the at the 21st century
17:54 I cipher Russia s it's been really good it's definitely special
17:58 a few things right up for us so yeah
18:01 got him huge yes
18:05 used be
18:11 his the no read them fun
18:18 I yeah I but not I go because I don't like
18:23 presumably you could buy dying know why you'd want to
18:27 his yeah but it might be better to look a
18:33 other things in that case I don't know I i'm not
18:36 just yeah good be presumably as possible I don't know I mean in theory can have
18:42 unlimited channels unlimited cues in Rabat
18:44 from what I've read asset perhaps you could have an unlimited TTL on something
18:48 up
18:48 or whatever and just never can shame it by I'd down
18:52 my it's sad if for instance gets which offer
18:56 whatever reason you might lose everything the hat say yep
19:00 yeah say okay many
19:04 bowling she was not called of
19:07 say gets right in ketchikan a reason youtube.com
19:11 not yet other like a lot I yet will waive
19:14 we for with my developer and I company who love snout
19:18 and he's he's coming had his he's implemented up a bunch of fat but JP I
19:22 stuff in item I just so it's natural to
19:24 continue on that from because we've we've already got and no tensions
19:29 running this stuff
19:30 so I we we go on top of want up what had already built
19:33 sir any other questions
19:37 got em those work low
19:41 upfront cost social and we are
19:44 a social media its API partner
19:48 sideways Sat we gave
19:51 at advertising agencies the ability it's ok to bulk upload
19:55 campaigns to Facebook and Twitter and I had to as well now
19:59 so so the total price but chose my allow a chipset manage maybe 500 campaigns at
20:04 once weekly doe
20:05 6,000 campaigns over one sinner and parents face
20:08 right think I set so I'm gonna
20:15 Hanover presumably to mark 10 are