Last active

Embed URL

HTTPS clone URL

SSH clone URL

You can clone with HTTPS or SSH.

Download Gist

stress testing the creation of a 1000 docker containers, comes across file descriptor and iptable errors

View stress-docker-run
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135
[] ubuntu@~$ sudo docker version
Client version: 0.6.3
Go version (client): go1.1.2
Git commit (client): b0a49a3
Server version: 0.6.3
Git commit (server): b0a49a3
Go version (server): go1.1.2
Last stable version: 0.6.4, please update docker
 
[] ubuntu@~$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 15962
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024000
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 15962
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
 
# hint from https://groups.google.com/forum/#!topic/docker-user/uWiC5R9CqEY
[] ubuntu@~$ cat /proc/sys/fs/inotify/max_user_watches
8192
 
[] ubuntu@~$ cat /stress.sh
#!/bin/bash
echo "Creating 1000 containers"
df -h .
for i in {1..1000}; do
id=`sudo docker run -d test/ssh /usr/sbin/sshd`
#echo "$i => $id" -n
echo -n "."
done
df -h .
 
[] ubuntu@~$ /stress.sh
Creating 1000 containers
Filesystem Size Used Avail Use% Mounted on
/dev/vda 40G 1.7G 36G 5% /
......................................................................................................................................................................................................................................................................................................................................................................................2013/10/18 12:57:45 Error: Error starting container da83e4a474ea: pipe2: too many open files
.2013/10/18 12:57:45 Error: Error starting container e13e6a60c69e: iptables failed: iptables -t nat -A DOCKER -p tcp --dport 50650 ! -i docker0 -j DNAT --to-destination 172.17.5.219:22
..2013/10/18 12:57:45 Error: Error starting container 41e1c9090d64: pipe2: too many open files
.2013/10/18 12:57:45 Error: Error starting container 3c2390c7ae54: iptables failed: iptables -t nat -A DOCKER -p tcp --dport 50653 ! -i docker0 -j DNAT --to-destination 172.17.5.222:22
.2013/10/18 12:57:45 Error: Error starting container 79b3976fc31c: iptables failed: iptables -t nat -A DOCKER -p tcp --dport 50654 ! -i docker0 -j DNAT --to-destination 172.17.5.223:22
.2013/10/18 12:57:46 Error: Error starting container 8a418403d944: iptables failed: iptables -t nat -A DOCKER -p tcp --dport 50655 ! -i docker0 -j DNAT --to-destination 172.17.5.224:22
.2013/10/18 12:57:46 Error: Error starting container 2112e187fe7f: iptables failed: iptables -t nat -A DOCKER -p tcp --dport 50656 ! -i docker0 -j DNAT --to-destination 172.17.5.225:22
.2013/10/18 12:57:46 Error: Error starting container ce5139c6c081: iptables failed: iptables -t nat -A DOCKER -p tcp --dport 50657 ! -i docker0 -j DNAT --to-destination 172.17.5.226:22
.2013/10/18 12:57:46 Error: Error starting container 85ee1f6c7922: iptables failed: iptables -t nat -A DOCKER -p tcp --dport 50658 ! -i docker0 -j DNAT --to-destination 172.17.5.227:22
.2013/10/18 12:57:46 Error: Error starting container 3e3dc60dee65: iptables failed: iptables -t nat -A DOCKER -p tcp --dport 50659 ! -i docker0 -j DNAT --to-destination 172.17.5.228:22
.2013/10/18 12:57:46 Error: Error starting container 1ec26be9e8a2: iptables failed: iptables -t nat -A DOCKER -p tcp --dport 50660 ! -i docker0 -j DNAT --to-destination 172.17.5.229:22
..................2013/10/18 12:57:49 Error: Error starting container 9d82210e63fc: pipe2: too many open files
...2013/10/18 12:57:49 Error: Error starting container f0c14d1c99f8: fork/exec /usr/bin/lxc-start: too many open files
.2013/10/18 12:57:49 Error: Error starting container a8afb1837e38: pipe2: too many open files
.2013/10/18 12:57:49 Error: Error starting container cf198bd6a0cb: iptables failed: iptables -t nat -A DOCKER -p tcp --dport 50683 ! -i docker0 -j DNAT --to-destination 172.17.5.252:22
.2013/10/18 12:57:49 unexpected EOF
.2013/10/18 12:57:49 unexpected EOF
.2013/10/18 12:57:49 Error: Error starting container bd0cfc17d22c: iptables failed: iptables -t nat -A DOCKER -p tcp --dport 50686 ! -i docker0 -j DNAT --to-destination 172.17.5.255:22
.2013/10/18 12:57:49 unexpected EOF
.....................2013/10/18 12:57:52 Error: Error starting container 40e1599d74bf: pipe2: too many open files
.2013/10/18 12:57:52 Error: Error starting container a3acd0dbdc54: iptables failed: iptables -t nat -A DOCKER -p tcp --dport 50709 ! -i docker0 -j DNAT --to-destination 172.17.6.22:22
.2013/10/18 12:57:53 Error: Error starting container 1dce00bc8f72: iptables failed: iptables -t nat -A DOCKER -p tcp --dport 50710 ! -i docker0 -j DNAT --to-destination 172.17.6.23:22
.2013/10/18 12:57:53 read unix /var/run/docker.sock: connection reset by peer
....2013/10/18 12:57:53 Error: Error starting container 10d95b1a1fe2: fork/exec /usr/bin/lxc-start: too many open files
..2013/10/18 12:57:53 Error: Error starting container 246f6d3eb751: pipe2: too many open files
.2013/10/18 12:57:53 unexpected EOF
.2013/10/18 12:57:53 unexpected EOF
.2013/10/18 12:57:53 Error: Error starting container 1b68396e5c89: pipe2: too many open files
.2013/10/18 12:57:54 unexpected EOF
.2013/10/18 12:57:54 Error: Error starting container d86ea259a1ea: iptables failed: iptables -t nat -A DOCKER -p tcp --dport 50721 ! -i docker0 -j DNAT --to-destination 172.17.6.34:22
.2013/10/18 12:57:54 unexpected EOF
.2013/10/18 12:57:54 unexpected EOF
.2013/10/18 12:57:54 unexpected EOF
.2013/10/18 12:57:54 Error: Error starting container 719321a68b47: iptables failed: iptables -t nat -A DOCKER -p tcp --dport 50725 ! -i docker0 -j DNAT --to-destination 172.17.6.38:22
.2013/10/18 12:57:54 Error: Error starting container 08eb459f2b89: iptables failed: iptables -t nat -A DOCKER -p tcp --dport 50726 ! -i docker0 -j DNAT --to-destination 172.17.6.39:22
.2013/10/18 12:57:54 Error: Error starting container b8f78a7cb7fd: iptables failed: iptables -t nat -A DOCKER -p tcp --dport 50727 ! -i docker0 -j DNAT --to-destination 172.17.6.40:22
.2013/10/18 12:57:54 Error: Error starting container 63f0a3057b0c: iptables failed: iptables -t nat -A DOCKER -p tcp --dport 50728 ! -i docker0 -j DNAT --to-destination 172.17.6.41:22
.2013/10/18 12:57:54 unexpected EOF
.2013/10/18 12:57:54 Error: Error starting container 8ac2aeab8b11: iptables failed: iptables -t nat -A DOCKER -p tcp --dport 50730 ! -i docker0 -j DNAT --to-destination 172.17.6.43:22
....2013/10/18 12:57:55 Error: Error starting container 10da41d68cba: pipe2: too many open files
.2013/10/18 12:57:55 Error: Error starting container f029e33f23f9: iptables failed: iptables -t nat -A DOCKER -p tcp --dport 50735 ! -i docker0 -j DNAT --to-destination 172.17.6.48:22
.2013/10/18 12:57:55 Error: Error starting container 54d7d0535c17: pipe2: too many open files
.2013/10/18 12:57:55 Error: Error starting container c1a50e08dcb8: fork/exec /usr/bin/lxc-start: too many open files
........2013/10/18 12:57:56 Error: Error starting container ccffe2fdf595: pipe2: too many open files
.2013/10/18 12:57:56 Error: Error starting container f5595aeec957: iptables failed: iptables -t nat -A DOCKER -p tcp --dport 50746 ! -i docker0 -j DNAT --to-destination 172.17.6.59:22
.2013/10/18 12:57:56 Error: Error starting container ce145463a442: iptables failed: iptables -t nat -A DOCKER -p tcp --dport 50747 ! -i docker0 -j DNAT --to-destination 172.17.6.60:22
.2013/10/18 12:57:57 Error: Error starting container 04567cd2d2f3: iptables failed: iptables -t nat -A DOCKER -p tcp --dport 50748 ! -i docker0 -j DNAT --to-destination 172.17.6.61:22
.2013/10/18 12:57:57 Error: Error starting container 57d217a3bb82: iptables failed: iptables -t nat -A DOCKER -p tcp --dport 50749 ! -i docker0 -j DNAT --to-destination 172.17.6.62:22
.2013/10/18 12:57:57 Error: Error starting container a2b8933531e5: iptables failed: iptables -t nat -A DOCKER -p tcp --dport 50750 ! -i docker0 -j DNAT --to-destination 172.17.6.63:22
.2013/10/18 12:57:57 unexpected EOF
.2013/10/18 12:57:57 Error: Error starting container 1af26c868a0c: iptables failed: iptables -t nat -A DOCKER -p tcp --dport 50752 ! -i docker0 -j DNAT --to-destination 172.17.6.65:22
.2013/10/18 12:57:57 unexpected EOF
.2013/10/18 12:57:57 unexpected EOF
.2013/10/18 12:57:57 unexpected EOF
.2013/10/18 12:57:57 Error: Error starting container 849223b54aa3: iptables failed: iptables -t nat -A DOCKER -p tcp --dport 50756 ! -i docker0 -j DNAT --to-destination 172.17.6.69:22
.2013/10/18 12:57:57 Error: Error starting container 7003a2e9d162: iptables failed: iptables -t nat -A DOCKER -p tcp --dport 50757 ! -i docker0 -j DNAT --to-destination 172.17.6.70:22
.2013/10/18 12:57:57 Error: Error starting container d743ecc0754d: iptables failed: iptables -t nat -A DOCKER -p tcp --dport 50758 ! -i docker0 -j DNAT --to-destination 172.17.6.71:22
.2013/10/18 12:57:57 Error: Error starting container 3afa52ed17a5: iptables failed: iptables -t nat -A DOCKER -p tcp --dport 50759 ! -i docker0 -j DNAT --to-destination 172.17.6.72:22
.2013/10/18 12:57:57 Error: Error starting container f3d41acce65f: iptables failed: iptables -t nat -A DOCKER -p tcp --dport 50760 ! -i docker0 -j DNAT --to-destination 172.17.6.73:22
.2013/10/18 12:57:57 Error: Error starting container 8b710126c54b: iptables failed: iptables -t nat -A DOCKER -p tcp --dport 50761 ! -i docker0 -j DNAT --to-destination 172.17.6.74:22
.2013/10/18 12:57:58 unexpected EOF
.2013/10/18 12:57:58 Error: Error starting container a923d8343807: iptables failed: iptables -t nat -A DOCKER -p tcp --dport 50763 ! -i docker0 -j DNAT --to-destination 172.17.6.76:22
.2013/10/18 12:57:58 unexpected EOF
.2013/10/18 12:57:58 Error: Error starting container 8f5b446472ea: iptables failed: iptables -t nat -A DOCKER -p tcp --dport 50765 ! -i docker0 -j DNAT --to-destination 172.17.6.78:22
.2013/10/18 12:57:58 unexpected EOF
.2013/10/18 12:57:58 Error: Error starting container 42e6b095787e: iptables failed: iptables -t nat -A DOCKER -p tcp --dport 50767 ! -i docker0 -j DNAT --to-destination 172.17.6.80:22
.2013/10/18 12:57:58 unexpected EOF
.2013/10/18 12:57:58 unexpected EOF
...........................................................................................................................................2013/10/18 12:58:22 unexpected EOF
..................................................................................................................................................2013/10/18 12:58:45 Error: Error starting container 59ad04fc96e1: iptables failed: iptables -t nat -A DOCKER -p tcp --dport 51054 ! -i docker0 -j DNAT --to-destination 172.17.7.111:22
............................................................................................................................................................................................................................Filesystem Size Used Avail Use% Mounted on
/dev/vda 40G 1.8G 36G 5% /
 
 
#
# here's how test/ssh was created
#
 
[] ubuntu@~$ cat Dockerfile
FROM base
EXPOSE 22
 
RUN sudo apt-get update
RUN dpkg-divert --local --rename --add /sbin/initctl && ln -s /bin/true /sbin/initctl
 
RUN sudo apt-get install -q -y openssh-server
RUN mkdir /var/run/sshd
 
[] ubuntu@~$ sudo docker build -t test/ssh .
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.