AK 3.6.2 System test results

ak-3.6.2-RC2-system-test.md

• Test run 1 :
  • https://jenkins.confluent.io/job/system-test-kafka-branch-builder/6104/
  • https://confluent-kafka-branch-builder-system-test-results-v2.s3-us-west-2.amazonaws.com/system-test-kafka-branch-builder--1711643683--apache--3.6--64f7a0a300/2024-03-28--001./2024-03-28--001./report.html
  • Failed Tests

Module: kafkatest.tests.core.group_mode_transactions_test
Class:  GroupModeTransactionsTest
Method: test_transactions
Arguments:
{
  "bounce_target": "clients",
  "failure_mode": "clean_bounce"
}


TimeoutError('Consumer consumed only 84056 out of 100000 messages in 90s')
Traceback (most recent call last):
  File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/tests/runner_client.py", line 184, in _do_run
    data = self.run_test()
  File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/tests/runner_client.py", line 262, in run_test
    return self.test_context.function(self.test)
  File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/mark/_mark.py", line 433, in wrapper
    return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/tests/kafkatest/tests/core/group_mode_transactions_test.py", line 290, in test_transactions
    num_messages_to_copy=self.num_seed_messages)
  File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/tests/kafkatest/tests/core/group_mode_transactions_test.py", line 252, in copy_messages_transactionally
    return self.drain_consumer(concurrent_consumer, num_messages_to_copy)
  File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/tests/kafkatest/tests/core/group_mode_transactions_test.py", line 214, in drain_consumer
    (len(consumer.messages_consumed[1]), num_messages, 90))
  File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/utils/util.py", line 58, in wait_until
    raise TimeoutError(err_msg() if callable(err_msg) else err_msg) from last_exception
ducktape.errors.TimeoutError: Consumer consumed only 84056 out of 100000 messages in 90s

Module: kafkatest.tests.core.mirror_maker_test
Class:  TestMirrorMakerService
Method: test_bounce
Arguments:
{
  "clean_shutdown": false,
  "security_protocol": "PLAINTEXT"
}


TimeoutError('')
Traceback (most recent call last):
  File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/tests/runner_client.py", line 184, in _do_run
    data = self.run_test()
  File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/tests/runner_client.py", line 262, in run_test
    return self.test_context.function(self.test)
  File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/mark/_mark.py", line 433, in wrapper
    return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/tests/kafkatest/tests/core/mirror_maker_test.py", line 170, in test_bounce
    self.run_produce_consume_validate(core_test_action=lambda: self.bounce(clean_shutdown=clean_shutdown))
  File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/tests/kafkatest/tests/produce_consume_validate.py", line 105, in run_produce_consume_validate
    core_test_action(*args)
  File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/tests/kafkatest/tests/core/mirror_maker_test.py", line 170, in <lambda>
    self.run_produce_consume_validate(core_test_action=lambda: self.bounce(clean_shutdown=clean_shutdown))
  File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/tests/kafkatest/tests/core/mirror_maker_test.py", line 106, in bounce
    wait_until(lambda: len(self.consumer.messages_consumed[1]) > num_consumed + 100, timeout_sec=60)
  File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/utils/util.py", line 58, in wait_until
    raise TimeoutError(err_msg() if callable(err_msg) else err_msg) from last_exception
ducktape.errors.TimeoutError

Module: kafkatest.tests.client.quota_test
Class:  QuotaTest
Method: test_quota
Arguments:
{
  "old_broker_throttling_behavior": true,
  "quota_type": "client-id"
}


ValueError('max() arg is an empty sequence')
Traceback (most recent call last):
  File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/tests/runner_client.py", line 184, in _do_run
    data = self.run_test()
  File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/tests/runner_client.py", line 262, in run_test
    return self.test_context.function(self.test)
  File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/mark/_mark.py", line 433, in wrapper
    return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/tests/kafkatest/tests/client/quota_test.py", line 169, in test_quota
    success, msg = self.validate(self.kafka, producer, consumer)
  File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/tests/kafkatest/tests/client/quota_test.py", line 197, in validate
    metric.value for k, metrics in producer.metrics(group='producer-metrics', name='outgoing-byte-rate', client_id=producer.client_id) for metric in metrics
ValueError: max() arg is an empty sequence

Module: kafkatest.tests.client.quota_test
Class:  QuotaTest
Method: test_quota
Arguments:
{
  "old_client_throttling_behavior": true,
  "quota_type": "client-id"
}


ValueError('max() arg is an empty sequence')
Traceback (most recent call last):
  File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/tests/runner_client.py", line 184, in _do_run
    data = self.run_test()
  File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/tests/runner_client.py", line 262, in run_test
    return self.test_context.function(self.test)
  File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/mark/_mark.py", line 433, in wrapper
    return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/tests/kafkatest/tests/client/quota_test.py", line 169, in test_quota
    success, msg = self.validate(self.kafka, producer, consumer)
  File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/tests/kafkatest/tests/client/quota_test.py", line 197, in validate
    metric.value for k, metrics in producer.metrics(group='producer-metrics', name='outgoing-byte-rate', client_id=producer.client_id) for metric in metrics
ValueError: max() arg is an empty sequence

Module: kafkatest.tests.client.quota_test
Class:  QuotaTest
Method: test_quota
Arguments:
{
  "override_quota": true,
  "quota_type": "(user, client-id)"
}


ValueError('max() arg is an empty sequence')
Traceback (most recent call last):
  File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/tests/runner_client.py", line 184, in _do_run
    data = self.run_test()
  File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/tests/runner_client.py", line 262, in run_test
    return self.test_context.function(self.test)
  File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/mark/_mark.py", line 433, in wrapper
    return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/tests/kafkatest/tests/client/quota_test.py", line 169, in test_quota
    success, msg = self.validate(self.kafka, producer, consumer)
  File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/tests/kafkatest/tests/client/quota_test.py", line 197, in validate
    metric.value for k, metrics in producer.metrics(group='producer-metrics', name='outgoing-byte-rate', client_id=producer.client_id) for metric in metrics
ValueError: max() arg is an empty sequence

Module: kafkatest.tests.client.quota_test
Class:  QuotaTest
Method: test_quota
Arguments:
{
  "consumer_num": 2,
  "quota_type": "client-id"
}


ValueError('max() arg is an empty sequence')
Traceback (most recent call last):
  File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/tests/runner_client.py", line 184, in _do_run
    data = self.run_test()
  File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/tests/runner_client.py", line 262, in run_test
    return self.test_context.function(self.test)
  File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/mark/_mark.py", line 433, in wrapper
    return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/tests/kafkatest/tests/client/quota_test.py", line 169, in test_quota
    success, msg = self.validate(self.kafka, producer, consumer)
  File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/tests/kafkatest/tests/client/quota_test.py", line 197, in validate
    metric.value for k, metrics in producer.metrics(group='producer-metrics', name='outgoing-byte-rate', client_id=producer.client_id) for metric in metrics
ValueError: max() arg is an empty sequence

Module: kafkatest.tests.client.quota_test
Class:  QuotaTest
Method: test_quota
Arguments:
{
  "override_quota": false,
  "quota_type": "client-id"
}

ValueError('max() arg is an empty sequence')
Traceback (most recent call last):
  File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/tests/runner_client.py", line 184, in _do_run
    data = self.run_test()
  File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/tests/runner_client.py", line 262, in run_test
    return self.test_context.function(self.test)
  File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/mark/_mark.py", line 433, in wrapper
    return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/tests/kafkatest/tests/client/quota_test.py", line 169, in test_quota
    success, msg = self.validate(self.kafka, producer, consumer)
  File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/tests/kafkatest/tests/client/quota_test.py", line 197, in validate
    metric.value for k, metrics in producer.metrics(group='producer-metrics', name='outgoing-byte-rate', client_id=producer.client_id) for metric in metrics
ValueError: max() arg is an empty sequence

Module: kafkatest.tests.client.quota_test
Class:  QuotaTest
Method: test_quota
Arguments:
{
  "override_quota": false,
  "quota_type": "(user, client-id)"
}

ValueError('max() arg is an empty sequence')
Traceback (most recent call last):
  File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/tests/runner_client.py", line 184, in _do_run
    data = self.run_test()
  File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/tests/runner_client.py", line 262, in run_test
    return self.test_context.function(self.test)
  File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/mark/_mark.py", line 433, in wrapper
    return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/tests/kafkatest/tests/client/quota_test.py", line 169, in test_quota
    success, msg = self.validate(self.kafka, producer, consumer)
  File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/tests/kafkatest/tests/client/quota_test.py", line 197, in validate
    metric.value for k, metrics in producer.metrics(group='producer-metrics', name='outgoing-byte-rate', client_id=producer.client_id) for metric in metrics
ValueError: max() arg is an empty sequence

Module: kafkatest.tests.client.quota_test
Class:  QuotaTest
Method: test_quota
Arguments:
{
  "override_quota": true,
  "quota_type": "client-id"
}

ValueError('max() arg is an empty sequence')
Traceback (most recent call last):
  File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/tests/runner_client.py", line 184, in _do_run
    data = self.run_test()
  File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/tests/runner_client.py", line 262, in run_test
    return self.test_context.function(self.test)
  File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/mark/_mark.py", line 433, in wrapper
    return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/tests/kafkatest/tests/client/quota_test.py", line 169, in test_quota
    success, msg = self.validate(self.kafka, producer, consumer)
  File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/tests/kafkatest/tests/client/quota_test.py", line 197, in validate
    metric.value for k, metrics in producer.metrics(group='producer-metrics', name='outgoing-byte-rate', client_id=producer.client_id) for metric in metrics
ValueError: max() arg is an empty sequence

Module: kafkatest.tests.client.quota_test
Class:  QuotaTest
Method: test_quota
Arguments:
{
  "override_quota": true,
  "quota_type": "user"
}

ValueError('max() arg is an empty sequence')
Traceback (most recent call last):
  File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/tests/runner_client.py", line 184, in _do_run
    data = self.run_test()
  File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/tests/runner_client.py", line 262, in run_test
    return self.test_context.function(self.test)
  File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/mark/_mark.py", line 433, in wrapper
    return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/tests/kafkatest/tests/client/quota_test.py", line 169, in test_quota
    success, msg = self.validate(self.kafka, producer, consumer)
  File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/tests/kafkatest/tests/client/quota_test.py", line 197, in validate
    metric.value for k, metrics in producer.metrics(group='producer-metrics', name='outgoing-byte-rate', client_id=producer.client_id) for metric in metrics
ValueError: max() arg is an empty sequence

Module: kafkatest.tests.client.quota_test
Class:  QuotaTest
Method: test_quota
Arguments:
{
  "override_quota": false,
  "quota_type": "user"
}

ValueError('max() arg is an empty sequence')
Traceback (most recent call last):
  File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/tests/runner_client.py", line 184, in _do_run
    data = self.run_test()
  File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/tests/runner_client.py", line 262, in run_test
    return self.test_context.function(self.test)
  File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/mark/_mark.py", line 433, in wrapper
    return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/tests/kafkatest/tests/client/quota_test.py", line 169, in test_quota
    success, msg = self.validate(self.kafka, producer, consumer)
  File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/tests/kafkatest/tests/client/quota_test.py", line 197, in validate
    metric.value for k, metrics in producer.metrics(group='producer-metrics', name='outgoing-byte-rate', client_id=producer.client_id) for metric in metrics
ValueError: max() arg is an empty sequence

Module: kafkatest.tests.core.network_degrade_test
Class:  NetworkDegradeTest
Method: test_latency
Arguments:
{
  "device_name": "eth0",
  "latency_ms": 50,
  "rate_limit_kbit": 1000,
  "task_name": "latency-100-rate-1000"
}

RemoteCommandError({'ssh_config': {'host': 'worker20', 'hostname': '10.140.38.82', 'user': 'ubuntu', 'port': 22, 'password': None, 'identityfile': '/home/jenkins/workspace/system-test-kafka-branch-builder/semaphore-muckrake.pem'}, 'hostname': 'worker20', 'ssh_hostname': '10.140.38.82', 'user': 'ubuntu', 'externally_routable_ip': '10.140.38.82', '_logger': <Logger kafkatest.tests.core.network_degrade_test.NetworkDegradeTest.test_latency.task_name=latency-100-rate-1000.device_name=eth0.latency_ms=50.rate_limit_kbit=1000-1268 (DEBUG)>, 'os': 'linux', '_ssh_client': <paramiko.client.SSHClient object at 0x7eff6c662f60>, '_sftp_client': <paramiko.sftp_client.SFTPClient object at 0x7eff6c682f98>, '_custom_ssh_exception_checks': None}, 'ping -i 1 -c 20 worker17', 1, b'')
Traceback (most recent call last):
  File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/tests/runner_client.py", line 184, in _do_run
    data = self.run_test()
  File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/tests/runner_client.py", line 262, in run_test
    return self.test_context.function(self.test)
  File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/mark/_mark.py", line 433, in wrapper
    return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/tests/kafkatest/tests/core/network_degrade_test.py", line 66, in test_latency
    for line in zk0.account.ssh_capture("ping -i 1 -c 20 %s" % zk1.account.hostname):
  File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/cluster/remoteaccount.py", line 680, in next
    return next(self.iter_obj)
  File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/cluster/remoteaccount.py", line 347, in output_generator
    raise RemoteCommandError(self, cmd, exit_status, stderr.read())
ducktape.cluster.remoteaccount.RemoteCommandError: ubuntu@worker20: Command 'ping -i 1 -c 20 worker17' returned non-zero exit status 1.

Module: kafkatest.tests.core.network_degrade_test
Class:  NetworkDegradeTest
Method: test_latency
Arguments:
{
  "device_name": "eth0",
  "latency_ms": 50,
  "rate_limit_kbit": 0,
  "task_name": "latency-100"
}

RemoteCommandError({'ssh_config': {'host': 'worker10', 'hostname': '10.140.51.240', 'user': 'ubuntu', 'port': 22, 'password': None, 'identityfile': '/home/jenkins/workspace/system-test-kafka-branch-builder/semaphore-muckrake.pem'}, 'hostname': 'worker10', 'ssh_hostname': '10.140.51.240', 'user': 'ubuntu', 'externally_routable_ip': '10.140.51.240', '_logger': <Logger kafkatest.tests.core.network_degrade_test.NetworkDegradeTest.test_latency.task_name=latency-100.device_name=eth0.latency_ms=50.rate_limit_kbit=0-1269 (DEBUG)>, 'os': 'linux', '_ssh_client': <paramiko.client.SSHClient object at 0x7eff6c6620f0>, '_sftp_client': <paramiko.sftp_client.SFTPClient object at 0x7eff6c6657f0>, '_custom_ssh_exception_checks': None}, 'ping -i 1 -c 20 worker6', 1, b'')
Traceback (most recent call last):
  File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/tests/runner_client.py", line 184, in _do_run
    data = self.run_test()
  File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/tests/runner_client.py", line 262, in run_test
    return self.test_context.function(self.test)
  File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/mark/_mark.py", line 433, in wrapper
    return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/tests/kafkatest/tests/core/network_degrade_test.py", line 66, in test_latency
    for line in zk0.account.ssh_capture("ping -i 1 -c 20 %s" % zk1.account.hostname):
  File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/cluster/remoteaccount.py", line 680, in next
    return next(self.iter_obj)
  File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/cluster/remoteaccount.py", line 347, in output_generator
    raise RemoteCommandError(self, cmd, exit_status, stderr.read())
ducktape.cluster.remoteaccount.RemoteCommandError: ubuntu@worker10: Command 'ping -i 1 -c 20 worker6' returned non-zero exit status 1.

• Test run 2 (Only failed tests above)
  • https://jenkins.confluent.io/job/system-test-kafka-branch-builder/6108/
  • https://confluent-kafka-branch-builder-system-test-results.s3-us-west-2.amazonaws.com/system-test-kafka-branch-builder--1711711413--apache--3.6--64f7a0a300/2024-03-29--001./2024-03-29--001./report.html
  • All the previously failed tests passesd in the second run