Skip to content

Instantly share code, notes, and snippets.

@XavM
Created March 25, 2016 13:59
Show Gist options
  • Save XavM/4c57a7455aedecfaaa7f to your computer and use it in GitHub Desktop.
Save XavM/4c57a7455aedecfaaa7f to your computer and use it in GitHub Desktop.
Pipe any thing to curl's stdin to be http requested, but "flush" curl's stdin every Nth request
## cUrl, used with "-K -", reads the config from the stdin, but waits for stdin's EOF before executing any HTTP requests
## See : http://comments.gmane.org/gmane.comp.web.curl.general/15001
## Using the split's filter options allows starting and piping a stream to a new curl's stdin every N requests
## A same connexion can be kept alive for those N requests, making perfs much much better
## TODO : Find a way to parallelise several curl in a "xargs -P" way
## Example : (the for loop is there to simulate some real stream, and to demonstrate the curl's required format)
N=1000
for i in {1..10000}; do
echo -en '
--next
--url "es1:9200/test/node'${i}'?ttl=30"
--header "Content-Type: application/json"
--data {"test":true}
\0
';
done \
| split -t '\0' -l ${N} --filter='curl -sSK -'
## You can even add some niceties and get some progress bar and a final report
N=1000
tStamp=$(date +%s)
for i in {1..10000}; do
echo -en '
--next
--url "es1:8084/test/node'${i}'?ttl=30"
--header "Content-Type: application/json"
--data {"test":true}
--output /dev/null
--write-out "%{http_code}\\n"
\0
';
done \
| split -u -t '\0' -l ${N} --filter='curl -sSK -' \
| pv -r -l \
| awk -v tStamp=${tStamp} '
{
rc[$0]++;
rc["total"]++;
}
END{
elapse = systime() - tStamp;
elapse = (elapse == 0) ? 1 : elapse;
print "-------------\nReport :";
print "\telapse : " elapse " Sec";
for(i in rc) {
print "\thttp_" i ":\t" rc[i] " | throughput: " rc[i] / elapse " Req/Sec";
};
}
'
## The split filter option is not that old, and could be unavailable on you system
## To build a fresh version of split from source :
#mkdir -p /tmp/coreutils
#cd /tmp/coreutils
#wget http://ftp.gnu.org/gnu/coreutils/coreutils-8.25.tar.xz
#yum install xz -y
#xz -d coreutils-8.25.tar.xz
#tar xvf coreutils-8.25.tar
#cd coreutils-8.25
#mkdir -p /tmp/coreutils/coreutils-8.25/output
#./configure --prefix /tmp/coreutils/coreutils-8.25/output
#make
#make install
#alias split="/tmp/coreutils/coreutils-8.25/output/bin/split"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment