Skip to content

Instantly share code, notes, and snippets.

@PegasusWang

PegasusWang/backend-architectures.md

Forked from ngocphamm/backend-architectures.md
Created April 28, 2018 07:48
Show Gist options
  • Save PegasusWang/91294caa0ab26a5c67b9b52d56178905 to your computer and use it in GitHub Desktop.
Save PegasusWang/91294caa0ab26a5c67b9b52d56178905 to your computer and use it in GitHub Desktop.
Download ZIP
Backend Architectures
Raw
backend-architectures.md

Backend Architectures

Twitter

ror, scala, jetty, erlang, thrift, mongrel, comet server, my-sql, memchached, varnish, kestrel(mq), starling, gizzard, cassandra, hadoop, vertica, munin, nagios, awstats

Facebook

php (with hiphop compiler), thrift, java(tomcat, jetty, minor), epoll, erlang, tornado, nodejs my-sql, memcahced, hadoop, hbase, hive, scribe(-hdfs), bigpipe, varnish, haystack, cassandra

Tumblr (Korean)

centos, sciapache, apache, nginx, (move out of)php, scala(selection), ruby, thrift, my-sql, redis, hbase, memcached, gearman, kafka, kestrel, finagle, varnish, ha-proxy, func, capistrano, puppet, jenkins

Quora

aws(ec2, s3), ubuntu, cloudfront, python, pylons, paste, tornado, thrift, comet server, memcached, haproxy, nginx

Pinterest

python, django, tornado, node.js, rabbitmq, nginx, haproxy, varnish, memcached, membase, redis, my-sql, mrjob, hadoop(elastic map reduce)

Instagram (Korean)

aws(s3, ebs), cloudfront, ubuntu, django(high-cpu extra-large), gunicorn, fabric, gearman, pyapns, twisted, postgre-sql(quadruple extra-large), mdadm(sofeware raid with ebs), repmgr, pgbouncer, redis, memcached, node2dm, munin, pingdom, pagerduty, sentry

Pulse

aws(ec2, s3, elb), tornado, scribe, mrjob, node-readability, haproxy, tornado, gae, mapreduce, django(appengine), google-cloud-storage, memcache, redis

Netflix

aws(ec2, s3, ebs, rds, dynamodb, sdb, sqs, sns, emr, elb, eip, vpc, direct-connect, iam), java(tomcat), mongodb, my-sql, casandra, hadoop, zookeeper, evcache, asgard, groovy, grails, zuul, priam and more netflix opensouces)

Spoqa

linux(2.6), nginx, uwsgi, aws(s3), dotcloud, mysql, redis, celery

StyleShare

ubuntu(12.04), aws(ec2, s3, elb), nginx, werkzeug, flask, postgre-sql, pgpool, memcached, gevent, celery, rabbitmq, fabric, boto, exceptional, flask-exceptional

Gae9(개9)

rabbitmq, celery, phash

TheBorn Coupon(더본쿠폰)

ubuntu(12.04), nhn ncloud, django, apache, mod_wsgi, ms-sql, memcached, fabric, south, wand, rsync, py-bcrypt, python-gcm, apns

Lezhin Comics(레진코믹스)

gae, "천만명 이하 규모는 구글 앱 엔진을 써도 충분하다.(Translate)" - @xguru

Cartoon Service(만화서비스) - Smartstudy

ubuntu(11.04) nbp (nhn business platform), my-sql(innodb), mongodb, django, fabric, uwsgi, nginx, memcached, cacti, npk, rsync, lftp

99designs

aws(ec2, s3, elb, rds, cloudwatch), varnish, php, ergo, my-sql, memcached, mongodb, redis, new-relic, statsd

TripAdvisor

linux, apache, java, tomcat, postgres-sql, lucene, velocity, memcached, jgroups, hadoop, cacti, nagios, custom[?]

  • supoorting infra: java, python, ruby, php, perl

Trello

nodejs, haproxy, redis, mongodb

GOV.UK

ubuntu(10.04), skyscape, akamai, puppet, puppetdb, nginx, unicorn, gunicorn, upstart, haproxy, varnish,

  • redirection: perl(manage and test), php(some), nodejs(side-by-side browser)
  • applications: ror, sinatra, scala, play(2.0), django, mapit
  • databases and other storage: mongodb, my-sql, postgres-sql, elasticsearch, solr, rabbitmq
  • monitoring, managing and alerting: statsd, logstash, ganglia, graphie, nagios
  • supporting tools: jenkins, new-relic, google-analytics, dyn, ses, font-forge, font-tools, zendesk, jekyll, heroku, clojure

StackExchange

windows server, sql-server, redis

YouPorn

php-frm, haproxy, activemq, varnish, redis, nginx, my-sql, syslog-ng, symfony2

LinkedIn

SoundCloud

Tagged

REFERENCES

INFRA, PLATFORMS, FRAMEWORKS

DATABASE, STORAGE, DATA-MINING

  • PostgreSQL: most advanced open source database.
  • repmgr: open source tools that helps DBAs and System administrators manage a cluster of PostgreSQL databases.
  • pgpool Wiki: middleware that works between PostgreSQL servers and a PostgreSQL database client.
  • PgBouncer: lightweight connection pooler for PostgreSQL.
  • SQLAlchemy: Python SQL toolkit and Object Relational Mapper that gives application developers the full power and flexibility of SQL.
  • South: intelligent schema and data migrations for ​Django projects.
  • twitter/gizzard · GitHub: flexible sharding framework for creating eventually-consistent distributed datastores
  • cassandra: used for high velocity writes, and lower velocity reads
  • hadoop: process unstructured and large datasets, hundreds of billions of rows.
  • vertica: used for analytics and large aggregations and joins so they don't have to write MapReduce jobs. (twitter)
  • mrjob: Run MapReduce jobs on Hadoop or Amazon Web Services
  • Apache Lucene - Welcome to Apache Lucene
  • Apache Solr: popular, blazing fast open source enterprise search platform from the Apache LuceneTM project
  • fatcache: Memcache on SSD.
  • google-cloud-storage: Store, access and manage your data on Google’s storage infrastructure. Take advantage of the scale and efficiency we have built over the years.
  • haystack: Facebook photo Infrastructure.
  • Netflix/EVCache: distributed in-memory data store for the cloud.
  • Elasticsearch: Open Source Distributed Real Time Search & Analytics
  • SQL-Server: Microsoft SQL Server.
  • Doctrine: PHP libraries primarily focused on providing persistence services

DEPLOY, MONITORING, UTILITIES

  • Fabric: library and command-line tool for streamlining the use of SSH for application deployment or systems administration tasks.
  • boto/boto · GitHub: python interface to aws
  • capistrano/capistrano · GitHub: Remote multi-server automation tool.
  • puppetlabs/puppet · GitHub: Server automation framework and application.
  • Exceptional: Exceptional tracks errors in web apps. It reports them in real-time.and gathers the info you need to fix them fast.
  • jzempel/flask-exceptional · GitHub: Exceptional extension for Flask.
  • rsync: utility that provides fast incremental file transfer.
  • Nagios: The Industry Standard in IT Infrastructure Monitoring
  • Munin: networked resource monitoring tool that can help analyze resource trends
  • AWStats: powerful and featureful tool that generates advanced web, streaming, ftp or mail server statistics, graphically
  • pingdom: Website monitoring. Monitor your server and network uptime and performance for free.
  • sentry: realtime error logging and aggregation platform
  • pagerduty: SaaS IT on-call schedule management, alerting and incident tracking.
  • scribe: server for aggregating log data that's streamed in realtime from clients. It is designed to be scalable and reliable
  • Sphinx: tool that makes it easy to create intelligent and beautiful
  • pHash.org: perceptual hash library
  • dahlia/wand · GitHub: The ctypes-based simple ImageMagick binding for Python.
  • py-bcrypt: strong password hashing for Python.
  • mdadm: manage MD devices aka Linux Software RAID.
  • twitter/snowflake · GitHub: network service for generating unique ID numbers at high scale with some simple guarantees
  • node-readability: Server side readability with node.js
  • Netflix/asgard: Web interface for application deployments and cloud management in Amazon Web Services (AWS)
  • Netflix/Priam: Co-Process for backup/recovery, Token Management, and Centralized Configuration management for Cassandra.
  • Netflix/zuul: edge service that provides dynamic routing, monitoring, resiliency, security, and more.
  • Cacti®: The Complete RRDTool-based Graphing Solution.
  • LFTP: sophisticated file transfer program.
  • lqez/npk: neat package system.
  • etsy/statsd: Simple daemon for easy stats aggregation.
  • new-relic: Application Performance Management & Monitoring.
  • Upstart - Wikipedia, the free encyclopedia
  • MapIt: map postcodes and geographical points to administrative areas
  • logstash: open source log management.
  • Ganglia: scalable distributed monitoring system for high-performance computing systems such as clusters and Grid.
  • Jenkins: extendable open source continuous integration server.
  • Graphite: Scalable Realtime Graphing.
  • Google-Analytics: Web Analytics & Reporting.
  • dyn: Managed DNS | Email Delivery | SMTP | Domain Registration
  • FontForge: outline font editor.
  • DTL FontTools: Dutch Type Library.
  • Zendesk.com: Customer Service Software.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment