Smart Answer test failures caused by lack of Internet connection

README.md

I disabled my Internet connection and ran the tests in the Smart Answers project to see how many relied on being able to make connections to the outside world.

test-output.txt

Chriss-MacBook-Air-2:~/Code/alphagov/smart-answers$ rake test 
  * DEFERRED: sequence of two questions a question raises an error should permit custom error messages per question. 
Run options: --seed 2460

# Running:

.......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................E...............................................................................................................E................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................E.EEEEE.................................E.....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................EEEE...............................................................................................................................................................................................................................................................................................................................................................................................................................EE...................................................EEE.....

Finished in 269.064189s, 14.5318 runs/s, 76.6880 assertions/s.

  1) Error:
CountryAndDateQuestionsTest#test_: with javascript should handle country and date questions. :
Capybara::Poltergeist::JavascriptError: One or more errors were raised in the Javascript code on the page. If you don't care about these errors, you can ignore them by setting js_errors: false in your Poltergeist configuration (see documentation for details).

ReferenceError: Can't find variable: $
ReferenceError: Can't find variable: $
    at http://127.0.0.1:51110/smartanswers/smart-answers.js:5
    test/integration/engine/country_and_date_questions_test.rb:23:in `block (2 levels) in <class:CountryAndDateQuestionsTest>'


  2) Error:
PrecalculationsTest#test_: with javascript should handle precalculations. :
Capybara::Poltergeist::JavascriptError: One or more errors were raised in the Javascript code on the page. If you don't care about these errors, you can ignore them by setting js_errors: false in your Poltergeist configuration (see documentation for details).

ReferenceError: Can't find variable: $
ReferenceError: Can't find variable: $
    at http://127.0.0.1:51110/smartanswers/smart-answers.js:5
    test/integration/engine/precalculations_test.rb:8:in `block (2 levels) in <class:PrecalculationsTest>'


  3) Error:
MoneyAndSalaryQuestionsTest#test_: with javascript should handle money and salary questions. :
Capybara::Poltergeist::JavascriptError: One or more errors were raised in the Javascript code on the page. If you don't care about these errors, you can ignore them by setting js_errors: false in your Poltergeist configuration (see documentation for details).

ReferenceError: Can't find variable: $
ReferenceError: Can't find variable: $
    at http://127.0.0.1:51110/smartanswers/smart-answers.js:5
    test/integration/engine/money_and_salary_questions_test.rb:8:in `block (2 levels) in <class:MoneyAndSalaryQuestionsTest>'


  4) Error:
ReportAProblemTest#test_: with javascript smart-answer outcome should have current url in report-a-problem. :
Capybara::Poltergeist::JavascriptError: One or more errors were raised in the Javascript code on the page. If you don't care about these errors, you can ignore them by setting js_errors: false in your Poltergeist configuration (see documentation for details).

ReferenceError: Can't find variable: $
ReferenceError: Can't find variable: $
    at http://127.0.0.1:51110/smartanswers/smart-answers.js:5
    test/integration/engine/report_a_problem_test.rb:8:in `block (3 levels) in <class:ReportAProblemTest>'


  5) Error:
ReportAProblemTest#test_: with javascript smart-answer outcome should show report-a-problem. :
Capybara::Poltergeist::JavascriptError: One or more errors were raised in the Javascript code on the page. If you don't care about these errors, you can ignore them by setting js_errors: false in your Poltergeist configuration (see documentation for details).

ReferenceError: Can't find variable: $
ReferenceError: Can't find variable: $
    at http://127.0.0.1:51110/smartanswers/smart-answers.js:5
    test/integration/engine/report_a_problem_test.rb:8:in `block (3 levels) in <class:ReportAProblemTest>'


  6) Error:
ReportAProblemTest#test_: with javascript smart-answer question should show report-a-problem. :
Capybara::Poltergeist::JavascriptError: One or more errors were raised in the Javascript code on the page. If you don't care about these errors, you can ignore them by setting js_errors: false in your Poltergeist configuration (see documentation for details).

ReferenceError: Can't find variable: $
ReferenceError: Can't find variable: $
    at http://127.0.0.1:51110/smartanswers/smart-answers.js:5
    test/integration/engine/report_a_problem_test.rb:8:in `block (3 levels) in <class:ReportAProblemTest>'


  7) Error:
ReportAProblemTest#test_: with javascript smart-answer start page should have current url in report-a-problem. :
Capybara::Poltergeist::JavascriptError: One or more errors were raised in the Javascript code on the page. If you don't care about these errors, you can ignore them by setting js_errors: false in your Poltergeist configuration (see documentation for details).

ReferenceError: Can't find variable: $
ReferenceError: Can't find variable: $
    at http://127.0.0.1:51110/smartanswers/smart-answers.js:5
    test/integration/engine/report_a_problem_test.rb:8:in `block (3 levels) in <class:ReportAProblemTest>'


  8) Error:
ReportAProblemTest#test_: with javascript smart-answer start page should show report-a-problem. :
Capybara::Poltergeist::JavascriptError: One or more errors were raised in the Javascript code on the page. If you don't care about these errors, you can ignore them by setting js_errors: false in your Poltergeist configuration (see documentation for details).

ReferenceError: Can't find variable: $
ReferenceError: Can't find variable: $
    at http://127.0.0.1:51110/smartanswers/smart-answers.js:5
    test/integration/engine/report_a_problem_test.rb:8:in `block (3 levels) in <class:ReportAProblemTest>'


  9) Error:
MultiChoiceAndValudQuestionsTest#test_: with javascript should handle multiple-choice and value questions. :
Capybara::Poltergeist::JavascriptError: One or more errors were raised in the Javascript code on the page. If you don't care about these errors, you can ignore them by setting js_errors: false in your Poltergeist configuration (see documentation for details).

ReferenceError: Can't find variable: $
ReferenceError: Can't find variable: $
    at http://127.0.0.1:51110/smartanswers/smart-answers.js:5
    test/integration/engine/multi_choice_and_value_questions_test.rb:8:in `block (2 levels) in <class:MultiChoiceAndValudQuestionsTest>'


 10) Error:
ChangingAnswerTest#test_: with javascript should be able to change checkbox answers. :
Capybara::Poltergeist::JavascriptError: One or more errors were raised in the Javascript code on the page. If you don't care about these errors, you can ignore them by setting js_errors: false in your Poltergeist configuration (see documentation for details).

ReferenceError: Can't find variable: $
ReferenceError: Can't find variable: $
    at http://127.0.0.1:51110/smartanswers/smart-answers.js:5
    test/integration/engine/changing_answer_test.rb:161:in `block (2 levels) in <class:ChangingAnswerTest>'


 11) Error:
ChangingAnswerTest#test_: with javascript should be able to change country and date answers. :
Capybara::Poltergeist::JavascriptError: One or more errors were raised in the Javascript code on the page. If you don't care about these errors, you can ignore them by setting js_errors: false in your Poltergeist configuration (see documentation for details).

ReferenceError: Can't find variable: $
ReferenceError: Can't find variable: $
    at http://127.0.0.1:51110/smartanswers/smart-answers.js:5
    test/integration/engine/changing_answer_test.rb:12:in `block (2 levels) in <class:ChangingAnswerTest>'


 12) Error:
ChangingAnswerTest#test_: with javascript should be able to change money and salary answers. :
Capybara::Poltergeist::JavascriptError: One or more errors were raised in the Javascript code on the page. If you don't care about these errors, you can ignore them by setting js_errors: false in your Poltergeist configuration (see documentation for details).

ReferenceError: Can't find variable: $
ReferenceError: Can't find variable: $
    at http://127.0.0.1:51110/smartanswers/smart-answers.js:5
    test/integration/engine/changing_answer_test.rb:59:in `block (2 levels) in <class:ChangingAnswerTest>'


 13) Error:
ChangingAnswerTest#test_: with javascript should be able to change value and multiple choice answers. :
Capybara::Poltergeist::JavascriptError: One or more errors were raised in the Javascript code on the page. If you don't care about these errors, you can ignore them by setting js_errors: false in your Poltergeist configuration (see documentation for details).

ReferenceError: Can't find variable: $
ReferenceError: Can't find variable: $
    at http://127.0.0.1:51110/smartanswers/smart-answers.js:5
    test/integration/engine/changing_answer_test.rb:97:in `block (2 levels) in <class:ChangingAnswerTest>'


 14) Error:
CheckboxQuestionsTest#test_: with javascript should allow selecting no options from a checkbox question. :
Capybara::Poltergeist::JavascriptError: One or more errors were raised in the Javascript code on the page. If you don't care about these errors, you can ignore them by setting js_errors: false in your Poltergeist configuration (see documentation for details).

ReferenceError: Can't find variable: $
ReferenceError: Can't find variable: $
    at http://127.0.0.1:51110/smartanswers/smart-answers.js:5
    test/integration/engine/checkbox_questions_test.rb:51:in `block (2 levels) in <class:CheckboxQuestionsTest>'


 15) Error:
CheckboxQuestionsTest#test_: with javascript should handle checkbox questions. :
Capybara::Poltergeist::JavascriptError: One or more errors were raised in the Javascript code on the page. If you don't care about these errors, you can ignore them by setting js_errors: false in your Poltergeist configuration (see documentation for details).

ReferenceError: Can't find variable: $
ReferenceError: Can't find variable: $
    at http://127.0.0.1:51110/smartanswers/smart-answers.js:5
    test/integration/engine/checkbox_questions_test.rb:8:in `block (2 levels) in <class:CheckboxQuestionsTest>'


 16) Error:
InputValidationTest#test_: with javascript should allow custom error messages with interpolation. :
Capybara::Poltergeist::JavascriptError: One or more errors were raised in the Javascript code on the page. If you don't care about these errors, you can ignore them by setting js_errors: false in your Poltergeist configuration (see documentation for details).

ReferenceError: Can't find variable: $
ReferenceError: Can't find variable: $
    at http://127.0.0.1:51110/smartanswers/smart-answers.js:5
    test/integration/engine/input_validation_test.rb:67:in `block (2 levels) in <class:InputValidationTest>'


 17) Error:
InputValidationTest#test_: with javascript should allow custom validation in calculations. :
Capybara::Poltergeist::JavascriptError: One or more errors were raised in the Javascript code on the page. If you don't care about these errors, you can ignore them by setting js_errors: false in your Poltergeist configuration (see documentation for details).

ReferenceError: Can't find variable: $
ReferenceError: Can't find variable: $
    at http://127.0.0.1:51110/smartanswers/smart-answers.js:5
    test/integration/engine/input_validation_test.rb:42:in `block (2 levels) in <class:InputValidationTest>'


 18) Error:
InputValidationTest#test_: with javascript should validate input and display errors. :
Capybara::Poltergeist::JavascriptError: One or more errors were raised in the Javascript code on the page. If you don't care about these errors, you can ignore them by setting js_errors: false in your Poltergeist configuration (see documentation for details).

ReferenceError: Can't find variable: $
ReferenceError: Can't find variable: $
    at http://127.0.0.1:51110/smartanswers/smart-answers.js:5
    test/integration/engine/input_validation_test.rb:9:in `block (2 levels) in <class:InputValidationTest>'

3910 runs, 20634 assertions, 0 failures, 18 errors, 0 skips
Coverage report Rcov style generated for Functional Tests to /Users/chrisroos/Code/alphagov/smart-answers/coverage/rcov
rake aborted!
Command failed with status (1): [ruby -I"lib:test" -I"/Users/chrisroos/Code/alphagov/smart-answers/.bundle/gems/gems/rake-10.4.2/lib" "/Users/chrisroos/Code/alphagov/smart-answers/.bundle/gems/gems/rake-10.4.2/lib/rake/rake_test_loader.rb" "test/functional/duplicate_flow_test.rb" "test/functional/routing_test.rb" "test/functional/smart_answers_controller_test.rb" "test/integration/engine/calendar_export_test.rb" "test/integration/engine/changing_answer_test.rb" "test/integration/engine/checkbox_questions_test.rb" "test/integration/engine/country_and_date_questions_test.rb" "test/integration/engine/data_partials_test.rb" "test/integration/engine/input_validation_test.rb" "test/integration/engine/money_and_salary_questions_test.rb" "test/integration/engine/multi_choice_and_value_questions_test.rb" "test/integration/engine/precalculations_test.rb" "test/integration/engine/report_a_problem_test.rb" "test/integration/healthcheck_test.rb" "test/integration/smart_answer_flows/additional_commodity_code_test.rb" "test/integration/smart_answer_flows/adoption_calculator_test.rb" "test/integration/smart_answer_flows/am_i_getting_minimum_wage_test.rb" "test/integration/smart_answer_flows/am_i_getting_minimum_wage_v2_test.rb" "test/integration/smart_answer_flows/appeal_a_benefits_decision_test.rb" "test/integration/smart_answer_flows/apply_for_probate_test.rb" "test/integration/smart_answer_flows/apply_tier_4_visa_test.rb" "test/integration/smart_answer_flows/benefit_cap_calculator_test.rb" "test/integration/smart_answer_flows/benefits_abroad_test.rb" "test/integration/smart_answer_flows/benefits_if_you_are_abroad_test.rb" "test/integration/smart_answer_flows/calculate_agricultural_holiday_entitlement_test.rb" "test/integration/smart_answer_flows/calculate_child_maintenance_test.rb" "test/integration/smart_answer_flows/calculate_married_couples_allowance_test.rb" "test/integration/smart_answer_flows/calculate_redundancy_pay_test.rb" "test/integration/smart_answer_flows/calculate_redundancy_pay_v3_test.rb" "test/integration/smart_answer_flows/calculate_state_pension_test.rb" "test/integration/smart_answer_flows/calculate_statutory_sick_pay_test.rb" "test/integration/smart_answer_flows/calculate_your_holiday_entitlement_test.rb" "test/integration/smart_answer_flows/check_uk_visa_test.rb" "test/integration/smart_answer_flows/check_uk_visa_v2_test.rb" "test/integration/smart_answer_flows/childcare_costs_for_tax_credits_test.rb" "test/integration/smart_answer_flows/energy_grants_calculator_test.rb" "test/integration/smart_answer_flows/estimate_self_assessment_penalties_test.rb" "test/integration/smart_answer_flows/find_a_british_embassy_test.rb" "test/integration/smart_answer_flows/help_if_you_are_arrested_abroad_test.rb" "test/integration/smart_answer_flows/inherits_someone_dies_without_will_test.rb" "test/integration/smart_answer_flows/legalisation_document_checker_test.rb" "test/integration/smart_answer_flows/marriage_abroad_test.rb" "test/integration/smart_answer_flows/marriage_abroad_v2_test.rb" "test/integration/smart_answer_flows/maternity_calculator_test.rb" "test/integration/smart_answer_flows/minimum_wage_calculator_employers_test.rb" "test/integration/smart_answer_flows/minimum_wage_calculator_employers_v2_test.rb" "test/integration/smart_answer_flows/overseas_passports_test.rb" "test/integration/smart_answer_flows/paternity_calculator_test.rb" "test/integration/smart_answer_flows/pip_checker_test.rb" "test/integration/smart_answer_flows/plan_adoption_leave_test.rb" "test/integration/smart_answer_flows/register_a_birth_test.rb" "test/integration/smart_answer_flows/register_a_death_test.rb" "test/integration/smart_answer_flows/request_for_flexible_working_test.rb" "test/integration/smart_answer_flows/simplified_expenses_checker_test.rb" "test/integration/smart_answer_flows/simplified_expenses_checker_v2_test.rb" "test/integration/smart_answer_flows/state_pension_through_partner_test.rb" "test/integration/smart_answer_flows/state_pension_topup_test.rb" "test/integration/smart_answer_flows/student_finance_calculator_test.rb" "test/integration/smart_answer_flows/towing_rules_test.rb" "test/integration/smart_answer_flows/uk_benefits_abroad_test.rb" "test/integration/smart_answer_flows/uk_benefits_abroad_v2_test.rb" "test/integration/smart_answer_flows/vat_payment_deadlines_test.rb" "test/integration/smartdown_flows/example_plugins_test.rb" "test/integration/smartdown_flows/question_types_test.rb" "test/unit/age_related_allowance_chooser_test.rb" "test/unit/calculators/agricultural_holiday_entitlement_calculator_test.rb" "test/unit/calculators/arrested_abroad_calculator_test.rb" "test/unit/calculators/birth_calculator_test.rb" "test/unit/calculators/child_maintenance_calculator_test.rb" "test/unit/calculators/childcare_cost_calculator_test.rb" "test/unit/calculators/commodity_code_calculator_test.rb" "test/unit/calculators/holiday_entitlement_test.rb" "test/unit/calculators/maternity_benefits_calculator_test.rb" "test/unit/calculators/maternity_paternity_calculator_test.rb" "test/unit/calculators/minimum_wage_calculator_test.rb" "test/unit/calculators/minimum_wage_calculator_v2_test.rb" "test/unit/calculators/night_work_hours_test.rb" "test/unit/calculators/passport_and_embassy_data_query_test.rb" "test/unit/calculators/pip_dates_test.rb" "test/unit/calculators/plan_adoption_leave_test.rb" "test/unit/calculators/plan_maternity_leave_test.rb" "test/unit/calculators/rates_query_test.rb" "test/unit/calculators/redundancy_calculator_test.rb" "test/unit/calculators/registrations_data_query_test.rb" "test/unit/calculators/registrations_data_query_v2_test.rb" "test/unit/calculators/self_assessment_penalties_test.rb" "test/unit/calculators/state_pension_amount_calculator_test.rb" "test/unit/calculators/state_pension_topup_calculator_test.rb" "test/unit/calculators/static_data_query_test.rb" "test/unit/calculators/statutory_sick_pay_calculator_test.rb" "test/unit/calculators/translator_links_test.rb" "test/unit/calculators/vat_payment_deadlines_test.rb" "test/unit/calculators/workplace_pension_calculator_test.rb" "test/unit/calendar_state_test.rb" "test/unit/calendar_test.rb" "test/unit/checkbox_question_test.rb" "test/unit/country_select_question_test.rb" "test/unit/date_helper_test.rb" "test/unit/date_question_test.rb" "test/unit/flow_registration_presenter_test.rb" "test/unit/flow_registry_test.rb" "test/unit/flow_test.rb" "test/unit/friendly_time_diff_test.rb" "test/unit/govspeak_presenter_test.rb" "test/unit/graph_presenter_test.rb" "test/unit/ics_renderer_test.rb" "test/unit/married_couple_allowance_calculator_test.rb" "test/unit/money_question_test.rb" "test/unit/money_test.rb" "test/unit/multiple_choice_question_test.rb" "test/unit/node_presenter_test.rb" "test/unit/optional_date_test.rb" "test/unit/phrase_list_test.rb" "test/unit/predicates_test.rb" "test/unit/question_base_test.rb" "test/unit/salary_question_test.rb" "test/unit/salary_test.rb" "test/unit/smartdown_adapter/plugin_factory_test.rb" "test/unit/smartdown_adapter/presenter_test.rb" "test/unit/smartdown_adapter/registry_test.rb" "test/unit/smartdown_adapter/text_question_presenter_test.rb" "test/unit/smartdown_content/smartdown_scenarios_test.rb" "test/unit/smartdown_plugins/animal-example-multiple_test.rb" "test/unit/smartdown_plugins/extendables/data_partial_test.rb" "test/unit/smartdown_plugins/landlord-immigration-check_test.rb" "test/unit/smartdown_plugins/pay-leave-for-parents-v2_test.rb" "test/unit/smartdown_plugins/pay-leave-for-parents_adoption_test.rb" "test/unit/smartdown_plugins/pay-leave-for-parents_test.rb" "test/unit/state_test.rb" "test/unit/value_question_test.rb" "test/unit/working_days_test.rb" "test/unit/world_location_test.rb" "test/unit/worldwide_organisation_test.rb" ]
/Users/chrisroos/Code/alphagov/smart-answers/lib/tasks/minitest.rake:12:in `block in <top (required)>'
Tasks: TOP => test:all
(See full trace by running task with --trace)