problem-description.md

Problem with JMeter metrics interpretation

I run load test using jmeter 2.8 on ubuntu trusty. You can read test description and results are here. JMeter configuration variables that I use are following:

export JMETER_RUMPUP=300
export JMETER_THREADS=5000
export JMETER_LOOPS=5000
export JMETER_DURATION=300
export LOAD_PATH=/fibonacci/15000

This creates following jmeter test plan. The idea is to create increasing load on the service that has autoscaling enabled and see how it is scaled.

At this moment I need to build plot with latency vs time. I use resulting jmeter.jtl file for that. This file has following data:

1452688120542|138|Dashing Demo|200|ThreadGroup 1-2|true|4301|2|2|138|1|0|null
1452688120438|241|Dashing Demo|200|ThreadGroup 1-1|true|4301|2|2|241|1|0|null
1452688120692|55|Dashing Demo|200|ThreadGroup 1-1|true|4301|2|2|55|1|0|null
1452688120692|55|Dashing Demo|200|ThreadGroup 1-2|true|4301|2|2|55|1|0|null
1452688120748|45|Dashing Demo|200|ThreadGroup 1-1|true|4301|2|2|45|1|0|null
1452688120748|62|Dashing Demo|200|ThreadGroup 1-2|true|4301|2|2|62|1|0|null
1452688120794|55|Dashing Demo|200|ThreadGroup 1-1|true|4301|2|2|55|1|0|null
...

The first collumn is timestamp in millisends (according to example from docs). So I guess I need to use it to display value of the latency (2nd collumn).

The problem is that the first column (timestamp) has level jumps. You can see this on this plot:

Time vs index (in array)

This plot is build in the following way using R:

read.csv('x86-jmeter.jtl', header = FALSE, sep = '|', as.is = TRUE) -> x86ResultsRaw
jpeg('test-1.jpg')
plot(x86ResultsRaw$V1, type='l', col='blue')

Time vs index (first 76.000 items)

If we look at first 76000 values of time, we will see acceptable image

This plot is build in the following way using R:

plot(x86ResultsRaw$V1[0:76000], type='l', col='blue')

You can see that this values are not sorted, this can be explained by difference of response time during different requests.

I re-run this test several tests on different environments and all tests had the same problems with tests.

Questions

The question are following:

1. context switches or internal jmeter algorithms?
2. what should be done to avoid this problems?

x86-jmeter.jmx

<?xml version="1.0" encoding="UTF-8"?>
<jmeterTestPlan version="1.2" properties="2.7" jmeter="2.12 r1636949">
  <hashTree>
    <TestPlan guiclass="TestPlanGui" testclass="TestPlan" testname="TestPlan" enabled="true">
      <stringProp name="TestPlan.comments"/>
      <boolProp name="TestPlan.functional_mode">false</boolProp>
      <boolProp name="TestPlan.serialize_threadgroups">false</boolProp>
      <elementProp name="TestPlan.user_defined_variables" elementType="Arguments" guiclass="ArgumentsPanel" testclass="Arguments" testname="TestPlan" enabled="true">
        <collectionProp name="Arguments.arguments"/>
      </elementProp>
      <stringProp name="TestPlan.user_define_classpath"/>
    </TestPlan>
    <hashTree>
      <ThreadGroup guiclass="ThreadGroupGui" testclass="ThreadGroup" testname="ThreadGroup" enabled="true">
        <stringProp name="ThreadGroup.on_sample_error">continue</stringProp>
        <elementProp name="ThreadGroup.main_controller" elementType="LoopController" guiclass="LoopControlPanel" testclass="LoopController" testname="ThreadGroup" enabled="true">
          <boolProp name="LoopController.continue_forever">false</boolProp>
          <intProp name="LoopController.loops">5000</intProp>
        </elementProp>
        <stringProp name="ThreadGroup.num_threads">5000</stringProp>
        <stringProp name="ThreadGroup.ramp_time">2500</stringProp>
        <longProp name="ThreadGroup.start_time">1453130821000</longProp>
        <longProp name="ThreadGroup.end_time">1453130821000</longProp>
        <boolProp name="ThreadGroup.scheduler">true</boolProp>
        <stringProp name="ThreadGroup.duration">301</stringProp>
        <stringProp name="ThreadGroup.delay"/>
        <boolProp name="ThreadGroup.delayedStart">true</boolProp>
      </ThreadGroup>
      <hashTree>
        <HTTPSamplerProxy guiclass="HttpTestSampleGui" testclass="HTTPSamplerProxy" testname="Dashing Demo" enabled="true">
          <elementProp name="HTTPsampler.Arguments" elementType="Arguments" guiclass="HTTPArgumentsPanel" testclass="Arguments" testname="Dashing Demo" enabled="true">
            <collectionProp name="Arguments.arguments"/>
          </elementProp>
          <stringProp name="HTTPSampler.domain">demo-cpu-checker.pcf.altoros.com</stringProp>
          <stringProp name="HTTPSampler.port">80</stringProp>
          <stringProp name="HTTPSampler.connect_timeout"/>
          <stringProp name="HTTPSampler.response_timeout"/>
          <stringProp name="HTTPSampler.protocol">http</stringProp>
          <stringProp name="HTTPSampler.contentEncoding"/>
          <stringProp name="HTTPSampler.path">/fibonacci/15050</stringProp>
          <stringProp name="HTTPSampler.method">GET</stringProp>
          <boolProp name="HTTPSampler.follow_redirects">true</boolProp>
          <boolProp name="HTTPSampler.auto_redirects">false</boolProp>
          <boolProp name="HTTPSampler.use_keepalive">true</boolProp>
          <boolProp name="HTTPSampler.DO_MULTIPART_POST">false</boolProp>
          <stringProp name="HTTPSampler.implementation"/>
          <boolProp name="HTTPSampler.monitor">false</boolProp>
          <stringProp name="HTTPSampler.embedded_url_re"/>
          <stringProp name="TestPlan.comments"/>
        </HTTPSamplerProxy>
        <hashTree/>
      </hashTree>
    </hashTree>
  </hashTree>
</jmeterTestPlan>