- You have a test that is failing occasionally locally or in jenkins
- It doesn't fail when you run just one test, it requires the entire test suite to run
Note that many of these instructions will be katello specific, but will work with slight modifications for foreman or other rails projects.
Whenever you run tests with minitest, it uses a random seed to make sure tests run in a different order every time. On a test run that is failing either locally or in jenkins look for something like:
Run options: -v --ci-reporter --seed 9192
In this example your seed number is 9192.
- Add 'minitest-bisect to your Gemfile (or bundler.d/my.local.rb)
bundle install
Here I've added MTB_VERBOSE=2 to provide more information,feel free to leave it off. I've provided the seed #, three include directories, and then a list of all files. This example assumes you are using bash:
shopt -s globstar
MTB_VERBOSE=2 minitest_bisect --seed 9192 -Itest/ -I../katello/test/ -I../katello/spec/ ../katello/spec/**/*.rb ../katello/test/**/*test.rb
This will run and attempt to find the smallest set of tests using this seed that will cause the test failure. In my experience it got it down to just three tests, when only two were required (but a huge improvement compared to over 1000 tests). When it is done, the output will looks somewhat like this:
/home/vagrant/.rvm/rubies/ruby-2.2.4/bin/ruby -Itest/ -I../katello/test/ -I../katello/spec/ -e 'require "./../katello/spec/helpers/katello_url_helper_spec.rb" ; require "./../katello/spec/helpers/locale_helper_methods.rb" ; require "./../katello/spec/helpers/organization_helper_methods.rb" ; require "./../katello/spec/helpers/product_helper_methods.rb" ; require "./../katello/spec/helpers/product_test_data.rb" ; require "./../katello/spec/helpers/repository_helper_methods.rb" ; require "./../katello/spec/helpers/repo_test_data.rb" ; require "./../katello/spec/helpers/sync_management_helper_spec.rb" ; require "./../katello/spec/lib/mapping_spec.rb" ; require "./../katello/spec/lib/model_util_spec.rb" ; require "./../katello/spec/lib/package_util_spec.rb" ; require "./../katello/spec/lib/url_matcher_spec.rb" ; require "./../katello/spec/models/activation_key_spec.rb" ; require "./../katello/spec/models/environment_spec.rb" ; require "./../katello/spec/models/glue/candlepin/repository_orchestration_spec.rb" ; require "./../katello/spec/models/gpg_key_spec.rb" ; require "./../katello/spec/models/host_collection_spec.rb" ; require "./../katello/spec/models/lazy_accessor_spec.rb" ; require "./../katello/spec/models/model_spec_helper.rb" ; require "./../katello/spec/models/organization_spec.rb" ; require "./../katello/spec/models/ping_spec.rb" ; require "./../katello/spec/models/product_spec.rb" ; require "./../katello/spec/models/provider_spec.rb" ; require "./../katello/spec/models/pulp_task_status_spec.rb" ; require "./../katello/spec/models/task_status_spec.rb" ; require "./../katello/spec/routing/rhsm/consumers_routing_spec.rb" ; require "./../katello/spec/support/memo_struct.rb" ; require "./../katello/spec/support/shared_examples/valid_tdl_shared_examples.rb" ; require "./../katello/test/actions/candlepin/activation_key_test.rb" ; require "./../katello/test/actions/candlepin/consumer_test.rb" ; require "./../katello/test/actions/candlepin/environment_test.rb" ; require "./../katello/test/actions/candlepin/listen_on_events_test.rb" ; require "./../katello/test/actions/candlepin/owner_test.rb" ; require "./../katello/test/actions/candlepin/product_test.rb" ; require "./../katello/test/actions/foreman/environment_test.rb" ; require "./../katello/test/actions/import_pool_test.rb" ; require "./../katello/test/actions/katello/activation_key_test.rb" ; require "./../katello/test/actions/katello/capsule_content_test.rb" ; require "./../katello/test/actions/katello/content_view_puppet_environment_test.rb" ; require "./../katello/test/actions/katello/content_view_puppet_module_test.rb" ; require "./../katello/test/actions/katello/content_view_test.rb" ; require "./../katello/test/actions/katello/content_view_version/destroy_test.rb" ; require "./../katello/test/actions/katello/content_view_version/republish_repositories_test.rb" ; require "./../katello/test/actions/katello/content_view_version_test.rb" ; require "./../katello/test/actions/katello/docker_repository_set_test.rb" ; require "./../katello/test/actions/katello/environment_test.rb" ; require "./../katello/test/actions/katello/event_queue_monitor_test.rb" ; require "./../katello/test/actions/katello/foreman_test.rb" ; require "./../katello/test/actions/katello/host/attach_subscriptions_test.rb" ; require "./../katello/test/actions/katello/host/auto_attach_subscription_test.rb" ; require "./../katello/test/actions/katello/host/destroy_test.rb" ; require "./../katello/test/actions/katello/host/erratum_test.rb" ; require "./../katello/test/actions/katello/host/generate_applicability_test.rb" ; require "./../katello/test/actions/katello/host/hypervisors_test.rb" ; require "./../katello/test/actions/katello/host/hypervisors_update_test.rb" ; require "./../katello/test/actions/katello/host/package_group_test.rb" ; require "./../katello/test/actions/katello/host/package_test.rb" ; require "./../katello/test/actions/katello/host/recalculate_errata_status_test.rb" ; require "./../katello/test/actions/katello/host/register_test.rb" ; require "./../katello/test/actions/katello/host/remove_subscriptions_test.rb" ; require "./../katello/test/actions/katello/host/unregister_test.rb" ; require "./../katello/test/actions/katello/host/update_content_overrides_test.rb" ; require "./../katello/test/actions/katello/host/update_test.rb" ; require "./../katello/test/actions/katello/host/upload_package_profile_test.rb" ; require "./../katello/test/actions/katello/organization_test.rb" ; require "./../katello/test/actions/katello/product_test.rb" ; require "./../katello/test/actions/katello/provider/update_test.rb" ; require "./../katello/test/actions/katello/repository/check_matching_content_test.rb" ; require "./../katello/test/actions/katello/repository/metadata_generate_test.rb" ; require "./../katello/test/actions/katello/repository_set_test.rb" ; require "./../katello/test/actions/katello/repository_test.rb" ; require "./../katello/test/actions/katello/sync_plan/add_products_test.rb" ; require "./../katello/test/actions/katello/sync_plan/remove_products_test.rb" ; require "./../katello/test/actions/katello/sync_plan/sync_plan_destroy_test.rb" ; require "./../katello/test/actions/pulp/consumer_test.rb" ; require "./../katello/test/actions/pulp/repository/refresh_test.rb" ; require "./../katello/test/actions/pulp/repository/remove_schedule_test.rb" ; require "./../katello/test/actions/pulp/repository/sync_progress_presenter_test.rb" ; require "./../katello/test/actions/pulp/repository/sync_progress_test.rb" ; require "./../katello/test/actions/pulp/repository/sync_test.rb" ; require "./../katello/test/actions/pulp/repository/update_schedule_test.rb" ; require "./../katello/test/actions/pulp/repository/upload_file_test.rb" ; require "./../katello/test/controllers/api/rhsm/candlepin_proxies_controller_test.rb" ; require "./../katello/test/controllers/api/v2/activation_keys_controller_test.rb" ; require "./../katello/test/controllers/api/v2/api_controller_test.rb" ; require "./../katello/test/controllers/api/v2/capsule_content_controller_test.rb" ; require "./../katello/test/controllers/api/v2/concerns/bulk_hosts_extensions_test.rb" ; require "./../katello/test/controllers/api/v2/content_uploads_controller_test.rb" ; require "./../katello/test/controllers/api/v2/content_view_components_controller_test.rb" ; require "./../katello/test/controllers/api/v2/content_view_filter_rules_controller_test.rb" ; require "./../katello/test/controllers/api/v2/content_view_filters_controller_test.rb" ; require "./../katello/test/controllers/api/v2/content_view_histories_controller_test.rb" ; require "./../katello/test/controllers/api/v2/content_view_puppet_modules_controller_test.rb" ; require "./../katello/test/controllers/api/v2/content_views_controller_test.rb" ; require "./../katello/test/controllers/api/v2/content_view_versions_controller_test.rb" ; require "./../katello/test/controllers/api/v2/docker_manifests_controller_test.rb" ; require "./../katello/test/controllers/api/v2/docker_tags_controller_test.rb" ; require "./../katello/test/controllers/api/v2/environments_controller_test.rb" ; require "./../katello/test/controllers/api/v2/errata_controller_test.rb" ; require "./../katello/test/controllers/api/v2/gpg_keys_controller_test.rb" ; require "./../katello/test/controllers/api/v2/host_autocomplete_controller_test.rb" ; require "./../katello/test/controllers/api/v2/host_collections_controller_test.rb" ; require "./../katello/test/controllers/api/v2/host_errata_controller_test.rb" ; require "./../katello/test/controllers/api/v2/hostgroups_controller_test.rb" ; require "./../katello/test/controllers/api/v2/host_packages_controller_test.rb" ; require "./../katello/test/controllers/api/v2/hosts_bulk_actions_controller_test.rb" ; require "./../katello/test/controllers/api/v2/hosts_controller_test.rb" ; require "./../katello/test/controllers/api/v2/host_subscriptions_controller_test.rb" ; require "./../katello/test/controllers/api/v2/organizations_controller_test.rb" ; require "./../katello/test/controllers/api/v2/ostree_branches_controller_test.rb" ; require "./../katello/test/controllers/api/v2/package_groups_controller_test.rb" ; require "./../katello/test/controllers/api/v2/packages_controller_test.rb" ; require "./../katello/test/controllers/api/v2/ping_controller_test.rb" ; require "./../katello/test/controllers/api/v2/products_bulk_actions_controller_test.rb" ; require "./../katello/test/controllers/api/v2/products_controller_test.rb" ; require "./../katello/test/controllers/api/v2/puppet_modules_controller_test.rb" ; require "./../katello/test/controllers/api/v2/repositories_bulk_actions_controller_test.rb" ; require "./../katello/test/controllers/api/v2/repositories_controller_test.rb" ; require "./../katello/test/controllers/api/v2/repository_sets_controller_test.rb" ; require "./../katello/test/controllers/api/v2/smart_proxies_controller_test.rb" ; require "./../katello/test/controllers/api/v2/subscriptions_controller_test.rb" ; require "./../katello/test/controllers/api/v2/sync_controller_test.rb" ; require "./../katello/test/controllers/api/v2/sync_plans_controller_test.rb" ; require "./../katello/test/controllers/auto_complete_search_controller_test.rb" ; require "./../katello/test/controllers/foreman/containers/steps_controller_test.rb" ; require "./../katello/test/controllers/foreman/hostgroups_controller_test.rb" ; require "./../katello/test/controllers/foreman/hosts_controller_test.rb" ; require "./../katello/test/controllers/foreman/organizations_controller_test.rb" ; require "./../katello/test/controllers/foreman/smart_proxies_controller_test.rb" ; require "./../katello/test/controllers/sync_management_controller_test.rb" ; require "./../katello/test/glue/candlepin/owner_test.rb" ; require "./../katello/test/glue/pulp/content_type_test.rb" ; require "./../katello/test/glue/pulp/repository_test.rb" ; require "./../katello/test/helpers/hosts_and_hostgroups_helper_test.rb" ; require "./../katello/test/lib/access_permissions_test.rb" ; require "./../katello/test/lib/capsule_content_test.rb" ; require "./../katello/test/lib/repo_discovery_test.rb" ; require "./../katello/test/lib/tasks/clean_backend_objects_test.rb" ; require "./../katello/test/lib/tasks/clean_published_repo_directories_test.rb" ; require "./../katello/test/lib/tasks/import_applicability_test.rb" ; require "./../katello/test/lib/tasks/regenerate_ueber_certs_test.rb" ; require "./../katello/test/lib/tasks/reimport_test.rb" ; require "./../katello/test/lib/tasks/repository_test.rb" ; require "./../katello/test/lib/tasks/seeds_test.rb" ; require "./../katello/test/lib/tasks/unify_hosts_test.rb" ; require "./../katello/test/lib/tasks/upgrades/3.0/update_subscription_facet_backend_data_test.rb" ; require "./../katello/test/lib/util/cdn_var_subsitutor_test.rb" ; require "./../katello/test/lib/util/docker_manifest_clause_generator_test.rb" ; require "./../katello/test/lib/util/model_test.rb" ; require "./../katello/test/lib/util/package_clause_generator_test.rb" ; require "./../katello/test/lib/util/package_test.rb" ; require "./../katello/test/lib/util/path_with_substitutions_test.rb" ; require "./../katello/test/lib/validators/content_view_erratum_filter_rule_validator_test.rb" ; require "./../katello/test/lib/validators/content_view_filter_version_validator_test.rb" ; require "./../katello/test/lib/validators/content_view_puppet_module_validator_test.rb" ; require "./../katello/test/lib/validators/katello_label_format_validator_test.rb" ; require "./../katello/test/lib/validators/katello_name_format_validator_test.rb" ; require "./../katello/test/mailers/errata_mailer_test.rb" ; require "./../katello/test/models/activation_key_test.rb" ; require "./../katello/test/models/association_test.rb" ; require "./../katello/test/models/authorization/activation_key_authorization_test.rb" ; require "./../katello/test/models/authorization/content_view_authorization_test.rb" ; require "./../katello/test/models/authorization/content_view_environment_authorization_test.rb" ; require "./../katello/test/models/authorization/content_view_history_authorization_test.rb" ; require "./../katello/test/models/authorization/content_view_version_authorization_test.rb" ; require "./../katello/test/models/authorization/gpg_key_authorization_test.rb" ; require "./../katello/test/models/authorization/host_collection_authorization_test.rb" ; require "./../katello/test/models/authorization/lifecycle_environment_authorization_test.rb" ; require "./../katello/test/models/authorization/organization_authorization_test.rb" ; require "./../katello/test/models/authorization/product_authorization_test.rb" ; require "./../katello/test/models/authorization/repository_authorization_test.rb" ; require "./../katello/test/models/concerns/container_extensions_test.rb" ; require "./../katello/test/models/concerns/content_facet_host_extensions_test.rb" ; require "./../katello/test/models/concerns/environment_extensions_test.rb" ; require "./../katello/test/models/concerns/hostgroup_extensions_test.rb" ; require "./../katello/test/models/concerns/host_managed_extensions_test.rb" ; require "./../katello/test/models/concerns/operatingsystem_extensions_test.rb" ; require "./../katello/test/models/concerns/redhat_extensions_test.rb" ; require "./../katello/test/models/concerns/smart_proxy_extensions_test.rb" ; require "./../katello/test/models/content_counts_calculator_test.rb" ; require "./../katello/test/models/content_override_test.rb" ; require "./../katello/test/models/content_view_component_test.rb" ; require "./../katello/test/models/content_view_docker_filter_rule_test.rb" ; require "./../katello/test/models/content_view_environment_test.rb" ; require "./../katello/test/models/content_view_erratum_filter_rule_test.rb" ; require "./../katello/test/models/content_view_filter_test.rb" ; require "./../katello/test/models/content_view_history_test.rb" ; require "./../katello/test/models/content_view_package_filter_rule_test.rb" ; require "./../katello/test/models/content_view_package_group_filter_rule_test.rb" ; require "./../katello/test/models/content_view_puppet_environment_test.rb" ; require "./../katello/test/models/content_view_puppet_module_test.rb" ; require "./../katello/test/models/content_view_test.rb" ; require "./../katello/test/models/content_view_version_test.rb" ; require "./../katello/test/models/docker_manifest_test.rb" ; require "./../katello/test/models/docker_tag_test.rb" ; require "./../katello/test/models/errata_status_test.rb" ; require "./../katello/test/models/erratum_test.rb" ; require "./../katello/test/models/file_unit_test.rb" ; require "./../katello/test/models/foreman/location_test.rb" ; require "./../katello/test/models/host_collection_test.rb" ; require "./../katello/test/models/host/content_facet_test.rb" ; require "./../katello/test/models/host/subscription_facet_test.rb" ; require "./../katello/test/models/kt_environment_test.rb" ; require "./../katello/test/models/label_from_name_test.rb" ; require "./../katello/test/models/organization_test.rb" ; require "./../katello/test/models/ostree_branch_test.rb" ; require "./../katello/test/models/package_group_test.rb" ; require "./../katello/test/models/ping_test.rb" ; require "./../katello/test/models/pool_test.rb" ; require "./../katello/test/models/pool_with_quantites_test.rb" ; require "./../katello/test/models/product_test.rb" ; require "./../katello/test/models/pulp_sync_status_test.rb" ; require "./../katello/test/models/puppet_module_test.rb" ; require "./../katello/test/models/repository_test.rb" ; require "./../katello/test/models/rhsm_fact_importer_test.rb" ; require "./../katello/test/models/rhsm_fact_parser_test.rb" ; require "./../katello/test/models/rpm_test.rb" ; require "./../katello/test/models/setting_test.rb" ; require "./../katello/test/models/subscription_status_test.rb" ; require "./../katello/test/models/subscription_test.rb" ; require "./../katello/test/models/sync_plan_test.rb" ; require "./../katello/test/models/trace_status_test.rb" ; require "./../katello/test/presenters/activation_key_subscriptions_presenter_test.rb" ; require "./../katello/test/presenters/content_view_version_compare_presenter_test.rb" ; require "./../katello/test/presenters/relation_presenter_test.rb" ; require "./../katello/test/presenters/repository_presenter_test.rb" ; require "./../katello/test/routing/candlepin_proxies_test.rb" ; require "./../katello/test/services/cert/rhsm_client_test.rb" ; require "./../katello/test/services/katello/candlepin/consumer_test.rb" ; require "./../katello/test/services/katello/event_queue_test.rb" ; require "./../katello/test/services/katello/pulp/erratum_test.rb" ; require "./../katello/test/services/katello/pulp/package_group_test.rb" ; require "./../katello/test/services/katello/pulp/puppet_module_test.rb" ; require "./../katello/test/services/katello/pulp/rpm_test.rb" ; require "./../katello/test/source_code_test.rb" ; require "./../katello/test/views/api/v2/docker_manifest_test.rb" ; require "./../katello/test/views/api/v2/erratum_test.rb" ; require "./../katello/test/views/api/v2/package_group_test.rb" ; require "./../katello/test/views/api/v2/rpm_test.rb"' -- --seed 9192 --server 32549 -n "/^(?:Katello::LocationsTest#(?:test_0002_create a default location on seed on a fresh install)|Katello::LocationTest::default locations#(?:test_0001_default location for subs or puppet cannot be destroyed))$/"
Final reproduction:
SEQUEL DEPRECATION WARNING: requiring 'sequel/no_core_ext' is deprecated and will be removed in Sequel 5. Just require 'sequel' instead.
/home/vagrant/.rvm/gems/ruby-2.2.4/gems/sequel-4.46.0/lib/sequel/no_core_ext.rb:4:in `<top (required)>'
/home/vagrant/.rvm/gems/ruby-2.2.4/gems/activesupport-4.2.8/lib/active_support/dependencies.rb:274:in `require'
/home/vagrant/.rvm/gems/ruby-2.2.4/gems/activesupport-4.2.8/lib/active_support/dependencies.rb:274:in `block in require'
/home/vagrant/.rvm/gems/ruby-2.2.4/gems/activesupport-4.2.8/lib/active_support/dependencies.rb:240:in `load_dependency'
/home/vagrant/.rvm/gems/ruby-2.2.4/gems/activesupport-4.2.8/lib/active_support/dependencies.rb:274:in `require'
/home/vagrant/.rvm/gems/ruby-2.2.4/gems/dynflow-0.8.23/lib/dynflow/persistence_adapters/sequel.rb:1:in `<top (required)>'
/home/vagrant/.rvm/gems/ruby-2.2.4/gems/activesupport-4.2.8/lib/active_support/dependencies.rb:274:in `require'
/home/vagrant/.rvm/gems/ruby-2.2.4/gems/activesupport-4.2.8/lib/active_support/dependencies.rb:274:in `block in require'
/home/vagrant/.rvm/gems/ruby-2.2.4/gems/activesupport-4.2.8/lib/active_support/dependencies.rb:240:in `load_dependency'
/home/vagrant/.rvm/gems/ruby-2.2.4/gems/activesupport-4.2.8/lib/active_support/dependencies.rb:274:in `require'
/home/vagrant/git/foreman/lib/foreman.rb:8: warning: already initialized constant Foreman::UUID_REGEXP
/home/vagrant/git/foreman/lib/foreman.rb:8: warning: previous definition of UUID_REGEXP was here
DEPRECATION WARNING: Sprockets method `register_engine` is deprecated.
Please register a mime type using `register_mime_type` then
use `register_compressor` or `register_transformer`.
https://github.com/rails/sprockets/blob/master/guides/extending_sprockets.md#supporting-all-versions-of-sprockets-in-processors
(called from block (2 levels) in <class:Engine> at /home/vagrant/.rvm/gems/ruby-2.2.4/gems/angular-rails-templates-0.1.2/lib/angular-rails-templates/engine.rb:47)
DEPRECATION WARNING: Sprockets method `register_engine` is deprecated.
Please register a mime type using `register_mime_type` then
use `register_compressor` or `register_transformer`.
https://github.com/rails/sprockets/blob/master/guides/extending_sprockets.md#supporting-all-versions-of-sprockets-in-processors
(called from block (2 levels) in <class:Engine> at /home/vagrant/.rvm/gems/ruby-2.2.4/gems/angular-rails-templates-0.1.2/lib/angular-rails-templates/engine.rb:47)
DEPRECATION WARNING: Sprockets method `register_engine` is deprecated.
Please register a mime type using `register_mime_type` then
use `register_compressor` or `register_transformer`.
https://github.com/rails/sprockets/blob/master/guides/extending_sprockets.md#supporting-all-versions-of-sprockets-in-processors
(called from block (2 levels) in <class:Engine> at /home/vagrant/.rvm/gems/ruby-2.2.4/gems/angular-rails-templates-0.1.2/lib/angular-rails-templates/engine.rb:47)
DEPRECATION WARNING: Sprockets method `register_engine` is deprecated.
Please register a mime type using `register_mime_type` then
use `register_compressor` or `register_transformer`.
https://github.com/rails/sprockets/blob/master/guides/extending_sprockets.md#supporting-all-versions-of-sprockets-in-processors
(called from block in <class:Engine> at /home/vagrant/.rvm/gems/ruby-2.2.4/gems/angular-rails-templates-0.1.2/lib/angular-rails-templates/engine.rb:51)
API controllers newer than Apipie cache! Run apipie:cache rake task to regenerate cache.
-- execute("SET CONSTRAINTS ALL DEFERRED;")
-> 0.0002s
Run options: --seed 9192 -n "/^(?:Katello::LocationsTest#(?:test_0002_create a default location on seed on a fresh install)|Katello::LocationTest::default locations#(?:test_0001_default location for subs or puppet cannot be destroyed))$/"
# Running:
.F
Finished in 4.522066s, 0.4423 runs/s, 1.1057 assertions/s.
1) Failure:
Katello::LocationTest::default locations#test_0001_default location for subs or puppet cannot be destroyed [/home/vagrant/git/katello/test/models/foreman/location_test.rb:20]:
Expected #<ActiveRecord::Relation []> to not be empty.
2 runs, 5 assertions, 1 failures, 0 errors, 0 skips
Coverage report Rcov style generated for MiniTest to /home/vagrant/git/foreman/coverage/rcov
Coverage report generated for MiniTest to /home/vagrant/git/foreman/coverage. 0.0 / 0.0 LOC (100.0%) covered.
This has providedb the test args needed to narrow it down to just a couple tests:
Run options: --seed 9192 -n "/^(?:Katello::LocationsTest#(?:test_0002_create a default location on seed on a fresh install)|Katello::LocationTest::default locations#(?:test_0001_default location for subs or puppet cannot be destroyed))$/"
In a rails environment this is not super simple. If you look at the very first line in the output above, you will see a SUPER long command that requires each test file and at then provides our run options at the end. You can copy and paste that full line to run this subset of tests that should reproduce the issue. Alternatively you may use this:
ruby -I../katello/test/ -Itest/ -I../katello/spec/ -e '(Dir["../katello/spec/**/*_spec.rb"] + Dir["../katello/test/**/*_test.rb"]).each{|f| print "#{f}\n"; require f}' -- $RUN_OPTIONS
so for the provided run options above, you'd run:
ruby -I../katello/test/ -Itest/ -I../katello/spec/ -e '(Dir["../katello/spec/**/*_spec.rb"] + Dir["../katello/test/**/*_test.rb"]).each{|f| require f}' -- --seed 9192 -n "/^(?:Katello::LocationsTest#(?:test_0002_create a default location on seed on a fresh install)|Katello::LocationTest::default locations#(?:test_0001_default location for subs or puppet cannot be destroyed))$/"
Note the '--' between before the run options. This will now run the handful of tests needed to reproduce your issue and hopefully help you more easily identify the problem.