Skip to content

Instantly share code, notes, and snippets.

@irvingpop

irvingpop/Ubuntu-1404.md

Last active August 29, 2015 14:11
Show Gist options
  • Save irvingpop/4fb621b866f3cf19942b to your computer and use it in GitHub Desktop.
Save irvingpop/4fb621b866f3cf19942b to your computer and use it in GitHub Desktop.
Download ZIP
Chef Server HA & DRBD performance tuning
Raw
rhel66-aws.md

backend systems:

  • c3.2xlarge - EBS optimized and Enhanced Networking enabled
  • DRBD backing volume: 80 GB SSD (ec2 ephemeral storage /dev/xvdb)
  • RHEL 6.6 HVM official image with updates as of 12/15/2014
  • DRBD 8.4.5-2 from elrepo

pgbenchin

[root@ip-rh-backend1 ~]# uname -a
Linux ip-rh-backend1.rhel.aws 2.6.32-504.1.3.el6.x86_64 #1 SMP Fri Oct 31 11:37:10 EDT 2014 x86_64 x86_64 x86_64 GNU/Linux
[root@ip-rh-backend1 ~]# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 6.6 (Santiago)

[root@ip-rh-backend1 ~]# ethtool eth0
Settings for eth0:
  Supported ports: [ ]
  Supported link modes:   10000baseT/Full 
  Supported pause frame use: No
  Supports auto-negotiation: No
  Advertised link modes:  Not reported
  Advertised pause frame use: No
  Advertised auto-negotiation: No
  Speed: 10000Mb/s
  Duplex: Full
  Port: Other
  PHYAD: 0
  Transceiver: Unknown!
  Auto-negotiation: off
  Current message level: 0x00000007 (7)
             drv probe link
  Link detected: yes

[root@ip-rh-backend1 ~]# ethtool -i eth0
driver: ixgbevf
version: 2.12.1-k
firmware-version: 
bus-info: 0000:00:03.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: yes
supports-priv-flags: no

eth0      Link encap:Ethernet  HWaddr 0A:CB:7A:AB:D0:A7  
          inet addr:33.33.33.223  Bcast:33.33.33.255  Mask:255.255.255.0
          inet6 addr: fe80::8cb:7aff:feab:d0a7/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:9001  Metric:1
          RX packets:93657727 errors:0 dropped:0 overruns:0 frame:0
          TX packets:94473196 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:23860467235 (22.2 GiB)  

[root@ip-rh-backend1 ~]# sudo -u opscode-pgsql /opt/opscode/embedded/bin/pgbench -c4 -C -s 100 -t 10000
Scale option ignored, using pgbench_branches table count = 1
starting vacuum...end.
transaction type: TPC-B (sort of)
scaling factor: 1
query mode: simple
number of clients: 4
number of threads: 1
number of transactions per client: 10000
number of transactions actually processed: 40000/40000
tps = 270.641451 (including connections establishing)
tps = 510.023661 (excluding connections establishing)

change to deadline:

[root@ip-rh-backend1 ~]# sudo -u opscode-pgsql /opt/opscode/embedded/bin/pgbench -c4 -C -s 100 -t 10000
Scale option ignored, using pgbench_branches table count = 1
starting vacuum...end.
transaction type: TPC-B (sort of)
scaling factor: 1
query mode: simple
number of clients: 4
number of threads: 1
number of transactions per client: 10000
number of transactions actually processed: 40000/40000
tps = 271.118580 (including connections establishing)
tps = 512.504639 (excluding connections establishing)

drbd tuning settings:

in: /var/opt/opscode/drbd/etc/pc0.res

  net {
    max-buffers 8000;
    max-epoch-size 8000;
    sndbuf-size 512k;
  }

  disk {
    disk-barrier no;
    disk-flushes no;
    al-extents 3389;
  }

then:
[root@ip-rh-backend1 ~]# sudo -u opscode-pgsql /opt/opscode/embedded/bin/pgbench -c4 -C -s 100 -t 10000
Scale option ignored, using pgbench_branches table count = 1
starting vacuum...end.
transaction type: TPC-B (sort of)
scaling factor: 1
query mode: simple
number of clients: 4
number of threads: 1
number of transactions per client: 10000
number of transactions actually processed: 40000/40000
tps = 268.101503 (including connections establishing)
tps = 505.494709 (excluding connections establishing)


[root@ip-rh-backend1 data]# dd if=/dev/zero of=testfile bs=1M count=10240
10240+0 records in
10240+0 records out
10737418240 bytes (11 GB) copied, 94.9498 s, 113 MB/s

disabling hugepages

echo never > /sys/kernel/mm/redhat_transparent_hugepage/enabled
echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag

[root@ip-rh-backend1 ~]# sudo -u opscode-pgsql /opt/opscode/embedded/bin/pgbench -c4 -C -s 100 -t 10000
Scale option ignored, using pgbench_branches table count = 1
starting vacuum...end.
transaction type: TPC-B (sort of)
scaling factor: 1
query mode: simple
number of clients: 4
number of threads: 1
number of transactions per client: 10000
number of transactions actually processed: 40000/40000
tps = 269.929699 (including connections establishing)
tps = 511.336915 (excluding connections establishing)

bonnie

Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
ip-rh-backen 29872M   667  98 94273   8 67016   8  3818  99 346505  19  3910 141
Latency             13003us     326ms     556ms    3005us   27354us    3267us
Version  1.96       ------Sequential Create------ --------Random Create--------
ip-rh-backend1.rhel -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 14497  59 +++++ +++ +++++ +++ 19990  82 +++++ +++ +++++ +++
Latency               113us     365us     412us     142us      27us      36us
1.96,1.96,ip-rh-backend1.rhel.aws,1,1419000518,29872M,,667,98,94273,8,67016,8,3818,99,346505,19,3910,141,16,,,,,14497,59,+++++,+++,+++++,+++,19990,82,+++++,+++,+++++,+++,13003us,326ms,556ms,3005us,27354us,3267us,113us,365us,412us,142us,27us,36us
Raw
Ubuntu-1404.md

backend systems:

  • c3.2xlarge - EBS optimized and Enhanced Networking enabled
  • DRBD backing volume: 80 GB SSD (ec2 ephemeral storage /dev/xvdb)
  • Ubuntu Trusty official release from 11/26/2014
  • DRBD 8.4.3 (packaged in Ubuntu kernel extras package)
root@ip-ub-backend1:~# uname -a
Linux ip-ub-backend1.trusty.aws 3.13.0-40-generic #69-Ubuntu SMP Thu Nov 13 17:53:56 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

root@ip-ub-backend1:~# ethtool eth0
Settings for eth0:
	Supported ports: [ ]
	Supported link modes:   10000baseT/Full 
	Supported pause frame use: No
	Supports auto-negotiation: No
	Advertised link modes:  Not reported
	Advertised pause frame use: No
	Advertised auto-negotiation: No
	Speed: 10000Mb/s
	Duplex: Full
	Port: Other
	PHYAD: 0
	Transceiver: Unknown!
	Auto-negotiation: off
	Current message level: 0x00000007 (7)
			       drv probe link
	Link detected: yes

root@ip-ub-backend1:~# ethtool -i eth0
driver: ixgbevf
version: 2.11.3-k
firmware-version: 
bus-info: 0000:00:03.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: yes
supports-priv-flags: no

root@ip-ub-backend1:~# ifconfig eth0
eth0      Link encap:Ethernet  HWaddr 0a:c5:b0:c7:b0:00  
          inet addr:33.33.33.186  Bcast:33.33.33.255  Mask:255.255.255.0
          inet6 addr: fe80::8c5:b0ff:fec7:b000/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:9001  Metric:1
          RX packets:4939261 errors:0 dropped:0 overruns:0 frame:0
          TX packets:10077242 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:1143441495 (1.1 GB)  TX bytes:69821363680 (69.8 GB)


root@ip-ub-backend1:~# sudo -u opscode-pgsql /opt/opscode/embedded/bin/pgbench -c4 -C -s 100 -t 10000
Scale option ignored, using pgbench_branches table count = 1
starting vacuum...end.
transaction type: TPC-B (sort of)
scaling factor: 1
query mode: simple
number of clients: 4
number of threads: 1
number of transactions per client: 10000
number of transactions actually processed: 40000/40000
tps = 278.693060 (including connections establishing)
tps = 538.669603 (excluding connections establishing)

deadline? already on deadline

hugepages

root@ip-ub-backend1:~# echo never > /sys/kernel/mm/transparent_hugepage/enabled 
root@ip-ub-backend1:~# echo never > /sys/kernel/mm/transparent_hugepage/defrag 
root@ip-ub-backend1:~# 
root@ip-ub-backend1:~# 
root@ip-ub-backend1:~# sudo -u opscode-pgsql /opt/opscode/embedded/bin/pgbench -c4 -C -s 100 -t 10000
Scale option ignored, using pgbench_branches table count = 1
starting vacuum...end.


transaction type: TPC-B (sort of)
scaling factor: 1
query mode: simple
number of clients: 4
number of threads: 1
number of transactions per client: 10000
number of transactions actually processed: 40000/40000
tps = 269.788173 (including connections establishing)
tps = 506.435555 (excluding connections establishing)

throughput

root@ip-ub-backend1:/var/opt/opscode/drbd/data# dd if=/dev/zero of=testfile bs=1M count=10240
10240+0 records in
10240+0 records out
10737418240 bytes (11 GB) copied, 69.7517 s, 154 MB/s

bonnie

Version  1.97       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
ip-ub-backen 30072M   902  99 120568   9 95652   7  5511  99 321545  12 +++++ +++
Latency              9547us    2176ms    2373ms    1778us   14175us     695us
Version  1.97       ------Sequential Create------ --------Random Create--------
ip-ub-backend1.trus -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
Latency                71us     251us     286us      66us      14us      37us
1.97,1.97,ip-ub-backend1.trusty.aws,1,1419000891,30072M,,902,99,120568,9,95652,7,5511,99,321545,12,+++++,+++,16,,,,,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,9547us,2176ms,2373ms,1778us,14175us,695us,71us,251us,286us,66us,14us,37us
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment