Cold page-load test-cycle notes

cold-test-cycle.md

From https://wiki.mozilla.org/TestEngineering/Performance/Raptor#Cold_Page-Load

Raptor cold page-load test process when running on Firefox/Chrome/Chromium desktop:

A new browser profile is created

__MAIN__

• calls raptor.run_tests - (https://searchfox.org/mozilla-central/rev/9b4b41b95cbcda63f565bdc24411e15248f91d83/testing/raptor/raptor/raptor.py#1875)

Perftest build_browser_profile - (https://searchfox.org/mozilla-central/rev/9b4b41b95cbcda63f565bdc24411e15248f91d83/testing/raptor/raptor/raptor.py#219)

• calls self.get_conditioned_profile - (https://searchfox.org/mozilla-central/source/testing/raptor/raptor/raptor.py#223)
• calls create_profile using self.conditioned_profile_dir - (https://searchfox.org/mozilla-central/rev/9b4b41b95cbcda63f565bdc24411e15248f91d83/testing/raptor/raptor/raptor.py#225)
  • calls self.profile.merge - (https://searchfox.org/mozilla-central/rev/9b4b41b95cbcda63f565bdc24411e15248f91d83/testing/raptor/raptor/raptor.py#233)
  • calls self.profile.set_preferences - (https://searchfox.org/mozilla-central/rev/9b4b41b95cbcda63f565bdc24411e15248f91d83/testing/raptor/raptor/raptor.py#238)
  • sets local_profile_dir to self.profile.profile - (https://searchfox.org/mozilla-central/rev/9b4b41b95cbcda63f565bdc24411e15248f91d83/testing/raptor/raptor/raptor.py#241)

Perftest get_conditioned_profile - (https://searchfox.org/mozilla-central/rev/9b4b41b95cbcda63f565bdc24411e15248f91d83/testing/raptor/raptor/raptor.py#181)

• makes temp_download_dir using tempfile.mkdtemp - (https://searchfox.org/mozilla-central/rev/9b4b41b95cbcda63f565bdc24411e15248f91d83/testing/raptor/raptor/raptor.py#186)
• calls get_current_platform - (https://searchfox.org/mozilla-central/rev/9b4b41b95cbcda63f565bdc24411e15248f91d83/testing/raptor/raptor/raptor.py#198)
• downloads conditioned profile into temp_download_dir, assigning to cond_prof_target_dir - (https://searchfox.org/mozilla-central/rev/9b4b41b95cbcda63f565bdc24411e15248f91d83/testing/raptor/raptor/raptor.py#200)
• sets self.conditioned_profile_dir by an os.path.join of temp_download_dir and cond_prof_target_dir - (https://searchfox.org/mozilla-central/rev/9b4b41b95cbcda63f565bdc24411e15248f91d83/testing/raptor/raptor/raptor.py#206)
• deletes temp_download_dir using shutil.rmtree - (https://searchfox.org/mozilla-central/rev/9b4b41b95cbcda63f565bdc24411e15248f91d83/testing/raptor/raptor/raptor.py#215)
• returns our self.conditioned_profile_dir

Perftest run_test_setup (https://searchfox.org/mozilla-central/source/testing/raptor/raptor/raptor.py#266)

• calls self.start_playback (https://searchfox.org/mozilla-central/source/testing/raptor/raptor/raptor.py#273)
  • gets playback config
  • calls self.playback.start - https://searchfox.org/mozilla-central/rev/9b4b41b95cbcda63f565bdc24411e15248f91d83/testing/raptor/raptor/raptor.py#273
  • calls self.set_browser_test_prefs - (https://searchfox.org/mozilla-central/rev/9b4b41b95cbcda63f565bdc24411e15248f91d83/testing/raptor/raptor/raptor.py#276)

Raptor run_test_setup - (https://searchfox.org/mozilla-central/rev/9b4b41b95cbcda63f565bdc24411e15248f91d83/testing/raptor/raptor/raptor.py#1035)

• generates test config - (https://searchfox.org/mozilla-central/rev/9b4b41b95cbcda63f565bdc24411e15248f91d83/testing/raptor/raptor/raptor.py#1045)
• calls self.install_raptor_webext - (https://searchfox.org/mozilla-central/rev/9b4b41b95cbcda63f565bdc24411e15248f91d83/testing/raptor/raptor/raptor.py#1056)

Browsertime run_test_setup - (https://searchfox.org/mozilla-central/rev/9b4b41b95cbcda63f565bdc24411e15248f91d83/testing/raptor/raptor/raptor.py#721)

• calls self.start_playback - (https://searchfox.org/mozilla-central/rev/9b4b41b95cbcda63f565bdc24411e15248f91d83/testing/raptor/raptor/raptor.py#731)

The desktop browser is started up

RaptorDesktop launch_desktop_browser - (https://searchfox.org/mozilla-central/rev/9b4b41b95cbcda63f565bdc24411e15248f91d83/testing/raptor/raptor/raptor.py#1226)

• raise NotImplementedError

RaptorDesktopFirefox launch_desktop_browser - (https://searchfox.org/mozilla-central/rev/9b4b41b95cbcda63f565bdc24411e15248f91d83/testing/raptor/raptor/raptor.py#1400)

• ... (release build and debug-mode stuff)
• calls self.start_runner_proc - (https://searchfox.org/mozilla-central/rev/9b4b41b95cbcda63f565bdc24411e15248f91d83/testing/raptor/raptor/raptor.py#1409)`

RaptorDesktop start_runner_proc (https://searchfox.org/mozilla-central/source/testing/raptor/raptor/raptor.py#1229)

• calls self.runner_start - (https://searchfox.org/mozilla-central/rev/9b4b41b95cbcda63f565bdc24411e15248f91d83/testing/raptor/raptor/raptor.py#1231)
• sets self.control_server.browser_proc (https://searchfox.org/mozilla-central/source/testing/raptor/raptor/raptor.py#1237)

Post-startup browser settle pause of 30 seconds (soon to be 1 sec, again)

A new tab is opened

The test URL is loaded; measurements taken

Perftest run_tests - (https://searchfox.org/mozilla-central/rev/9b4b41b95cbcda63f565bdc24411e15248f91d83/testing/raptor/raptor/raptor.py#286)

• calls self.run_test - (https://searchfox.org/mozilla-central/rev/9b4b41b95cbcda63f565bdc24411e15248f91d83/testing/raptor/raptor/raptor.py#290)
• calls self.run_test_teardown - (https://searchfox.org/mozilla-central/rev/9b4b41b95cbcda63f565bdc24411e15248f91d83/testing/raptor/raptor/raptor.py#295)
• returns self.process_results- (https://searchfox.org/mozilla-central/rev/9b4b41b95cbcda63f565bdc24411e15248f91d83/testing/raptor/raptor/raptor.py#296)
• calls self.clean_up (finally; only called one (1) time, on ) - (https://searchfox.org/mozilla-central/rev/9b4b41b95cbcda63f565bdc24411e15248f91d83/testing/raptor/raptor/raptor.py#298)

RaptorDesktop run_test - (https://searchfox.org/mozilla-central/source/testing/raptor/raptor/raptor.py#1239)

• calls self.run_test_cold - (https://searchfox.org/mozilla-central/rev/9b4b41b95cbcda63f565bdc24411e15248f91d83/testing/raptor/raptor/raptor.py#1268)

RaptorDesktop run_test_cold - (https://searchfox.org/mozilla-central/rev/9b4b41b95cbcda63f565bdc24411e15248f91d83/testing/raptor/raptor/raptor.py#1292)

• [browser_cycle == 1] - 1st cold page-load browser cycle:
  • calls self.run_test_setup - (https://searchfox.org/mozilla-central/source/testing/raptor/raptor/raptor.py#1320)
  • calls self.launch_desktop_browser - (https://searchfox.org/mozilla-central/source/testing/raptor/raptor/raptor.py#1335)
  • sets self.control_server._finished = False
  • calls self.wait_for_test_finish - (https://searchfox.org/mozilla-central/source/testing/raptor/raptor/raptor.py#1343)
• [browser_cycle >= 2] - 2nd+ cold page-load browser cycles:
  • calls self.build_browser_profile - (https://searchfox.org/mozilla-central/source/testing/raptor/raptor/raptor.py#1330)
  • calls self.runner.profile = self.profile to update our runner's profile - (https://searchfox.org/mozilla-central/source/testing/raptor/raptor/raptor.py#1333)
  • calls self.run_test_setup - (https://searchfox.org/mozilla-central/source/testing/raptor/raptor/raptor.py#1335)
  • calls self.launch_desktop_browser - (https://searchfox.org/mozilla-central/source/testing/raptor/raptor/raptor.py#1335)
  • calls self.wait_for_test_finish - (https://searchfox.org/mozilla-central/source/testing/raptor/raptor/raptor.py#1343)

The tab is closed

The desktop browser is shut down

Perftest run_test_teardown - https://searchfox.org/mozilla-central/rev/9b4b41b95cbcda63f565bdc24411e15248f91d83/testing/raptor/raptor/raptor.py#305

• calls self.check_for_crashes - (https://searchfox.org/mozilla-central/rev/9b4b41b95cbcda63f565bdc24411e15248f91d83/testing/raptor/raptor/raptor.py#306)
• calls self.gecko_profiler_symbolicate - (https://searchfox.org/mozilla-central/rev/9b4b41b95cbcda63f565bdc24411e15248f91d83/testing/raptor/raptor/raptor.py#310)
• calls self.gecko_profiler_clean - (https://searchfox.org/mozilla-central/rev/9b4b41b95cbcda63f565bdc24411e15248f91d83/testing/raptor/raptor/raptor.py#313)

Raptor run_test_teardown - (https://searchfox.org/mozilla-central/rev/9b4b41b95cbcda63f565bdc24411e15248f91d83/testing/raptor/raptor/raptor.py#1104)

• calls self.playback.stop - (https://searchfox.org/mozilla-central/rev/9b4b41b95cbcda63f565bdc24411e15248f91d83/testing/raptor/raptor/raptor.py#761) then
• sets self.playback = None - (https://searchfox.org/mozilla-central/rev/9b4b41b95cbcda63f565bdc24411e15248f91d83/testing/raptor/raptor/raptor.py#1108-1109)
• calls self.remove_raptor_webext - (https://searchfox.org/mozilla-central/rev/9b4b41b95cbcda63f565bdc24411e15248f91d83/testing/raptor/raptor/raptor.py#1111)

RaptorDesktop run_test_teardown - (https://searchfox.org/mozilla-central/rev/9b4b41b95cbcda63f565bdc24411e15248f91d83/testing/raptor/raptor/raptor.py#1359)

• calls self.runner.stop - (https://searchfox.org/mozilla-central/rev/9b4b41b95cbcda63f565bdc24411e15248f91d83/testing/raptor/raptor/raptor.py#1363)
• calls self.remove_raptor_webext - (https://searchfox.org/mozilla-central/rev/9b4b41b95cbcda63f565bdc24411e15248f91d83/testing/raptor/raptor/raptor.py#1111)

Browsertime - run_test_teardown - (https://searchfox.org/mozilla-central/rev/9b4b41b95cbcda63f565bdc24411e15248f91d83/testing/raptor/raptor/raptor.py#756)

• calls self.playback.stop then
• sets self.playback = None - (https://searchfox.org/mozilla-central/rev/9b4b41b95cbcda63f565bdc24411e15248f91d83/testing/raptor/raptor/raptor.py#761-762)

Raptor - remove_raptor_webext- (https://searchfox.org/mozilla-central/rev/9b4b41b95cbcda63f565bdc24411e15248f91d83/testing/raptor/raptor/raptor.py#1154)

Entire process is repeated for the remaining browser cycles (25 cycles total)

The measurements from all browser cycles are used to calculate overall results

Perftest clean_up - (https://searchfox.org/mozilla-central/rev/9b4b41b95cbcda63f565bdc24411e15248f91d83/testing/raptor/raptor/raptor.py#335-336)

• pass

Raptor clean_up - (https://searchfox.org/mozilla-central/rev/9b4b41b95cbcda63f565bdc24411e15248f91d83/testing/raptor/raptor/raptor.py#1169)

• clears control-server's waits - (https://searchfox.org/mozilla-central/rev/9b4b41b95cbcda63f565bdc24411e15248f91d83/testing/raptor/raptor/raptor.py#1173)
  • calls self.control_server.stop - (https://searchfox.org/mozilla-central/rev/9b4b41b95cbcda63f565bdc24411e15248f91d83/testing/raptor/raptor/raptor.py#1175

RaptorDesktop clean_up - (https://searchfox.org/mozilla-central/rev/9b4b41b95cbcda63f565bdc24411e15248f91d83/testing/raptor/raptor/raptor.py#1380)

• calls self.runner.stop

Browsertime - clean_up - (https://searchfox.org/mozilla-central/rev/9b4b41b95cbcda63f565bdc24411e15248f91d83/testing/raptor/raptor/raptor.py#767)

• calls super(Browsertime, self).clean_up() - https://searchfox.org/mozilla-central/rev/9b4b41b95cbcda63f565bdc24411e15248f91d83/testing/raptor/raptor/raptor.py#768