Skip to content

Instantly share code, notes, and snippets.

@ragingwind
Last active February 21, 2024 14:40
Star You must be signed in to star a gist
Save ragingwind/5840075 to your computer and use it in GitHub Desktop.
Backend Architectures Keywords and References

Backend Architectures

ror, scala, jetty, erlang, thrift, mongrel, comet server, my-sql, memchached, varnish, kestrel(mq), starling, gizzard, cassandra, hadoop, vertica, munin, nagios, awstats

php (with hiphop compiler), thrift, java(tomcat, jetty, minor), epoll, erlang, tornado, nodejs my-sql, memcahced, hadoop, hbase, hive, scribe(-hdfs), bigpipe, varnish, haystack, cassandra

centos, sciapache, apache, nginx, (move out of)php, scala(selection), ruby, thrift, my-sql, redis, hbase, memcached, gearman, kafka, kestrel, finagle, varnish, ha-proxy, func, capistrano, puppet, jenkins

aws(ec2, s3), ubuntu, cloudfront, python, pylons, paste, tornado, thrift, comet server, memcached, haproxy, nginx

python, django, tornado, node.js, rabbitmq, nginx, haproxy, varnish, memcached, membase, redis, my-sql, mrjob, hadoop(elastic map reduce)

aws(s3, ebs), cloudfront, ubuntu, django(high-cpu extra-large), gunicorn, fabric, gearman, pyapns, twisted, postgre-sql(quadruple extra-large), mdadm(sofeware raid with ebs), repmgr, pgbouncer, redis, memcached, node2dm, munin, pingdom, pagerduty, sentry

aws(ec2, s3, elb), tornado, scribe, mrjob, node-readability, haproxy, tornado, gae, mapreduce, django(appengine), google-cloud-storage, memcache, redis

aws(ec2, s3, ebs, rds, dynamodb, sdb, sqs, sns, emr, elb, eip, vpc, direct-connect, iam), java(tomcat), mongodb, my-sql, casandra, hadoop, zookeeper, evcache, asgard, groovy, grails, zuul, priam and more netflix opensouces)

linux(2.6), nginx, uwsgi, aws(s3), dotcloud, mysql, redis, celery

ubuntu(12.04), aws(ec2, s3, elb), nginx, werkzeug, flask, postgre-sql, pgpool, memcached, gevent, celery, rabbitmq, fabric, boto, exceptional, flask-exceptional

rabbitmq, celery, phash

ubuntu(12.04), nhn ncloud, django, apache, mod_wsgi, ms-sql, memcached, fabric, south, wand, rsync, py-bcrypt, python-gcm, apns

gae, "천만명 이하 규모는 구글 앱 엔진을 써도 충분하다.(Translate)" - @xguru

ubuntu(11.04) nbp (nhn business platform), my-sql(innodb), mongodb, django, fabric, uwsgi, nginx, memcached, cacti, npk, rsync, lftp

aws(ec2, s3, elb, rds, cloudwatch), varnish, php, ergo, my-sql, memcached, mongodb, redis, new-relic, statsd

linux, apache, java, tomcat, postgres-sql, lucene, velocity, memcached, jgroups, hadoop, cacti, nagios, custom[?]

  • supoorting infra: java, python, ruby, php, perl

nodejs, haproxy, redis, mongodb

ubuntu(10.04), skyscape, akamai, puppet, puppetdb, nginx, unicorn, gunicorn, upstart, haproxy, varnish,

  • redirection: perl(manage and test), php(some), nodejs(side-by-side browser)
  • applications: ror, sinatra, scala, play(2.0), django, mapit
  • databases and other storage: mongodb, my-sql, postgres-sql, elasticsearch, solr, rabbitmq
  • monitoring, managing and alerting: statsd, logstash, ganglia, graphie, nagios
  • supporting tools: jenkins, new-relic, google-analytics, dyn, ses, font-forge, font-tools, zendesk, jekyll, heroku, clojure

windows server, sql-server, redis

php-frm, haproxy, activemq, varnish, redis, nginx, my-sql, syslog-ng, symfony2

REFERENCES

INFRA, PLATFORMS, FRAMEWORKS

DATABASE, STORAGE, DATA-MINING

  • PostgreSQL: most advanced open source database.
  • repmgr: open source tools that helps DBAs and System administrators manage a cluster of PostgreSQL databases.
  • pgpool Wiki: middleware that works between PostgreSQL servers and a PostgreSQL database client.
  • PgBouncer: lightweight connection pooler for PostgreSQL.
  • SQLAlchemy: Python SQL toolkit and Object Relational Mapper that gives application developers the full power and flexibility of SQL.
  • South: intelligent schema and data migrations for ​Django projects.
  • twitter/gizzard · GitHub: flexible sharding framework for creating eventually-consistent distributed datastores
  • cassandra: used for high velocity writes, and lower velocity reads
  • hadoop: process unstructured and large datasets, hundreds of billions of rows.
  • vertica: used for analytics and large aggregations and joins so they don't have to write MapReduce jobs. (twitter)
  • mrjob: Run MapReduce jobs on Hadoop or Amazon Web Services
  • Apache Lucene - Welcome to Apache Lucene
  • Apache Solr: popular, blazing fast open source enterprise search platform from the Apache LuceneTM project
  • fatcache: Memcache on SSD.
  • google-cloud-storage: Store, access and manage your data on Google’s storage infrastructure. Take advantage of the scale and efficiency we have built over the years.
  • haystack: Facebook photo Infrastructure.
  • Netflix/EVCache: distributed in-memory data store for the cloud.
  • Elasticsearch: Open Source Distributed Real Time Search & Analytics
  • SQL-Server: Microsoft SQL Server.
  • Doctrine: PHP libraries primarily focused on providing persistence services

DEPLOY, MONITORING, UTILITIES

  • Fabric: library and command-line tool for streamlining the use of SSH for application deployment or systems administration tasks.
  • boto/boto · GitHub: python interface to aws
  • capistrano/capistrano · GitHub: Remote multi-server automation tool.
  • puppetlabs/puppet · GitHub: Server automation framework and application.
  • Exceptional: Exceptional tracks errors in web apps. It reports them in real-time.and gathers the info you need to fix them fast.
  • jzempel/flask-exceptional · GitHub: Exceptional extension for Flask.
  • rsync: utility that provides fast incremental file transfer.
  • Nagios: The Industry Standard in IT Infrastructure Monitoring
  • Munin: networked resource monitoring tool that can help analyze resource trends
  • AWStats: powerful and featureful tool that generates advanced web, streaming, ftp or mail server statistics, graphically
  • pingdom: Website monitoring. Monitor your server and network uptime and performance for free.
  • sentry: realtime error logging and aggregation platform
  • pagerduty: SaaS IT on-call schedule management, alerting and incident tracking.
  • scribe: server for aggregating log data that's streamed in realtime from clients. It is designed to be scalable and reliable
  • Sphinx: tool that makes it easy to create intelligent and beautiful
  • pHash.org: perceptual hash library
  • dahlia/wand · GitHub: The ctypes-based simple ImageMagick binding for Python.
  • py-bcrypt: strong password hashing for Python.
  • mdadm: manage MD devices aka Linux Software RAID.
  • twitter/snowflake · GitHub: network service for generating unique ID numbers at high scale with some simple guarantees
  • node-readability: Server side readability with node.js
  • Netflix/asgard: Web interface for application deployments and cloud management in Amazon Web Services (AWS)
  • Netflix/Priam: Co-Process for backup/recovery, Token Management, and Centralized Configuration management for Cassandra.
  • Netflix/zuul: edge service that provides dynamic routing, monitoring, resiliency, security, and more.
  • Cacti®: The Complete RRDTool-based Graphing Solution.
  • LFTP: sophisticated file transfer program.
  • lqez/npk: neat package system.
  • etsy/statsd: Simple daemon for easy stats aggregation.
  • new-relic: Application Performance Management & Monitoring.
  • Upstart - Wikipedia, the free encyclopedia
  • MapIt: map postcodes and geographical points to administrative areas
  • logstash: open source log management.
  • Ganglia: scalable distributed monitoring system for high-performance computing systems such as clusters and Grid.
  • Jenkins: extendable open source continuous integration server.
  • Graphite: Scalable Realtime Graphing.
  • Google-Analytics: Web Analytics & Reporting.
  • dyn: Managed DNS | Email Delivery | SMTP | Domain Registration
  • FontForge: outline font editor.
  • DTL FontTools: Dutch Type Library.
  • Zendesk.com: Customer Service Software.
@runehvalsoe
Copy link

Nice a very usefull list - I am missing a link to Graph Databases (NoSQL) like http://www.neo4j.org/ etc.

@lindaliukas
Copy link

Thank you! I've used this for so many projects :)

@lazyjin
Copy link

lazyjin commented Feb 2, 2016

Greate help! Thank you

@endalamaw1
Copy link

You are managing a web
app, built on django framework. Nginx is used as a reverse proxy and hosted on Ubuntu
14.04 server. It uses celery as a task processor, for sending email and downloading
files. The app uses redis as a caching server and PostgreSQL as the main database.
For reporting purposes, the app fetches aggregate data from PostgreSQL and stores in
Elasticsearch. Kafka is used for real-time messaging.
a. You are getting an INTERNAL SERVER ERROR, while accessing the main page
of the app. (a) What might be the possible source of the issue? (b) What
procedure will you follow to fix the issue?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment