phund

Finding a Ruby memory leak using a time analysis

When developing a program in Ruby, you may sometimes encounter a memory leak. For a while now, Ruby has a facility to gather information about what objects are laying around: ObjectSpace.

There are several approaches one can take to debug a leak. This discusses a time-based approach, where a full memory dump is generated every, say, 5 minutes, during a time that the memory leak is showing up. Afterwards, one can look at all the objects, and find out which ones are staying around, causing the

phund

Testing and learning

http://hack.darkn3ss.com/

http://link-base.org/

http://ringzer0team.com/

http://www.irongeek.com/

phund

Process List

SELECT (NOW() - query_start) as duration, pid, usename, application_name, client_addr, client_port, backend_start, query_start,
wait_event, wait_event_type, state, query
FROM pg_stat_activity 
WHERE pid <> pg_backend_pid()
AND state <> 'idle'
ORDER BY query_start ASC;

phund

# Create ext4 RAID

sudo mdadm --create --verbose /dev/md0 --level=10 --raid-devices=4 \
   --name=mysql /dev/sdb /dev/sdc /dev/sdd /dev/sde
sudo mkdir /mysql-ext4
sudo mkfs.ext4 /dev/md0
sudo mount -o rw,relatime /dev/md0 /mysql-ext4
sudo mkdir /mysql-ext4/msb
sudo chown revin.revin /mysql-ext4/msb

phund

# YOU MAY SKIP THIS FILE

# Suppress diff for generated files
build/* linguist-generated=true

phund

Tuning ZFS + Postgres to outperform EXT4 + Postgres

Please refer to ZFS 2-3x slower than EXT4 to see how ZFS defaults + Postgres defaults severely underperform EXT4 defaults + Postgres defaults (and also to know more about the system on which these benchmarks were performed). This page documents how to tune ZFS + Postgres to give better performance for the tpcb-like benchmark.

BIG FAT WARNING

Please do not copy these settings blindly because I am myself not clear on why/how these settings had the impact they did. For example, I cannot explain why full_page_writes=off independently did not give that much boost, nor did an optimized PG configuration. However, putting both of them together gave a 2-4x boost compared to baseline numbers.

Benchmark results

phund

source 'https://rubygems.org'

gem 'mechanize'

phund

-- show running queries (pre 9.2)
SELECT procpid, age(clock_timestamp(), query_start), usename, current_query 
FROM pg_stat_activity 
WHERE current_query != '<IDLE>' AND current_query NOT ILIKE '%pg_stat_activity%' 
ORDER BY query_start desc;

-- show running queries (9.2)
SELECT pid, age(clock_timestamp(), query_start), usename, query 
FROM pg_stat_activity 
WHERE query != '<IDLE>' AND query NOT ILIKE '%pg_stat_activity%' 

phund

For a project I am on I need to use a key-value store to converts file-paths to fixnum IDs. The dataset will typically be in the range of 100 000 to 1 000 000. These tests use 305 000 file paths to fixnum IDs.

The Different Key-Value stores tested are:

Daybreak: "Daybreak is a simple and very fast key value store for ruby" GDBM: GNU dbm. "a simple database engine for storing key-value pairs on disk." DBM: "The DBM class provides a wrapper to a Unix-style dbm or Database Manager library" PStore: "PStore implements a file based persistence mechanism based on a Hash. "

Out of these, all except Daybreak are in the Ruby standard library.

phund

history.replaceState && history.replaceState(
  null, '', location.pathname + location.search.replace(/[\?&]message=[^&]+/, '').replace(/^&/, '?')
);