Jeff-Lewis

Backend Architectures

Twitter

ror, scala, jetty, erlang, thrift, mongrel, comet server, my-sql, memchached, varnish, kestrel(mq), starling, gizzard, cassandra, hadoop, vertica, munin, nagios, awstats

• Introduce Gizzard of Twitter - 트위터의 새로운 분산 관리 라이브러리 Gizzard를 소개합니다.
• The Architecture Twitter Uses to Deal with 150M Active Users, 300K QPS, a 22 MB/S Firehose, and Send Tweets in Under 5 Seconds
• Scaling Twitter: Making Twitter 10000 Percent Faster

Jeff-Lewis

function isIterator(obj)
{
   return obj&&obj.next&&obj.throw;
}

function isRunWithContext(obj)
{
   return obj&&typeof obj=='function'&&obj.ctx;
}

Jeff-Lewis

I've developed a useful feature in KeystoneJS that lets you populate a relationship from either side, while only storing the data on one side, and am looking for feedback on whether it is something that could / should be brought back into mongoose itself. (It might be possible to add as a separate package but I suspect there'd be too much rewriting of mongoose internals for that to be a good idea).

I've added this as an issue in mongoose for consideration: #1888 but am leaving this gist in place because the examples are easier to read.

I've used Posts and Categories as a basic, contrived example to demonstrate what I'm talking about here; in reality you'd rarely load all the posts for a category but there are other real world cases where it's less unreasonable you'd want to do this, and Posts + Categories is an easy way to demo it.

The problem

The built-in population feature is really useful; not just for

Jeff-Lewis

var Model = require('mongoose').Model;


function softRemove(schema) {
  
  if (!schema.path('isRemoved')) {
    schema.add({ isRemoved : { type: Boolean, index: true, default: false } });
  }
  
  if (!schema.path('removedAt')) {

Jeff-Lewis

var moment = require('moment');
var tz = require('timezone/loaded');
var time = require('time');  

module.exports = function() {
  var strftime_format = '%F %T %z'; // used to convert a date into a normalized strftime format with timezone
  var moment_format = 'YYYY-MM-DD HH:mm:ss zz'; // moment.js LDML format for parsing date strings

  /**
   * Convert a Javascript Date into node-time wrapper with the appropriate timezone.

Jeff-Lewis

'use strict';
const asyncWrap = process.binding('async_wrap');
const fs = require('fs');

//
// Track
//
// 1. Setup a global variable to track the context of the async_hook_init_function
let callbackContext = 'root';
process.nextTick(function () {

Jeff-Lewis

"use strict";
var clsModule = require("continuation-local-storage");
const superagent = require("superagent");
const assert = require("assert");

var http = require("http");

var keepAlive = process.env.KEEP_ALIVE !== "0";
var httpAgent = new http.Agent({
  keepAlive: keepAlive,

Jeff-Lewis

@mafintosh asks: "Does anyone have a good code example of when to use setImmediate instead of nextTick?"

https://twitter.com/mafintosh/status/624590818125352960

The answer is "generally anywhere outside of core".

process.nextTick is barely asynchronous. Flow-wise it is asynchronous, but it will trigger before any other asynchronous events can (timers, io, etc.) and thus can starve the event loop.

In this script I show a starved event loop where I just synchronously block, use nextTick and setImmediate

Jeff-Lewis

domain Module Postmortem

Usability Issues

Implicit Behavior

It's possible for a developer to create a new domain and then simply run domain.enter(). Which then acts as a catch-all for any exception in the future that couldn't be observed by the thrower. Allowing a module author to intercept the exceptions of unrelated code in a different module. Preventing

Jeff-Lewis

// context
var mongoose = require('mongoose');
mongoose.connect('mongodb://...');

/**
 * Generates Mongoose uniqueness validator
 * 
 * @param string modelName
 * @param string field
 * @param boolean caseSensitive