I'm attempting to reproduce the errors described in 21061, but can't locally.
One difference I note is Travis runs Safari 12.0.0 and I'm running 12.0.3.
Below is the output I get from running
./node_modules/.bin/gulp test --safari --files=test/unit/test-performance.js Note that the tests that failed in Travis CI pass in my run:
PeformanceObserver metrics
should forward paint metrics for performance entries
● created after performance service registered
should forward first input metrics for performance entries
● created after performance service registered
and some other tests fail instead?
ericnyc-macbookpro:amphtml ericnyc$ ./node_modules/.bin/gulp test --safari --files=test/unit/test-performance.js
[11:18:29] Using gulpfile ~/git/amphtml/gulpfile.js
[11:18:29] Starting 'update-packages'...
[11:18:31] Installed lint rules from build-system/eslint-rules
[11:18:31] All packages in node_modules are up to date.
[11:18:31] Finished 'update-packages' after 1.72 s
[11:18:31] Starting 'build'...
[11:18:31] Running build. Press Ctrl + C to cancel...
[11:18:31] To skip building during future test runs, use --nobuild with your gulp test command.
[11:18:31] Building the runtime for local testing with the prod AMP config.
[11:18:31] ⤷ Use --config={canary|prod} with your gulp build command to specify which config to apply.
[11:18:31] Building all AMP extensions.
[11:18:31] ⤷ Use --noextensions to skip building extensions.
[11:18:31] ⤷ Use --extensions=amp-foo,amp-bar to choose which extensions to build.
[11:18:31] ⤷ Use --extensions=minimal_set to build just the extensions needed to load article.amp.html.
[11:18:31] ⤷ Use --extensions_from=examples/foo.amp.html to build extensions from example docs.
[11:18:32] Recompiled CSS in amp.css (1.481 s)
[11:18:32] Recompiled CSS in video-autoplay.css (022 ms)
[11:18:33] Recompiled CSS in amp-access (672 ms)
[11:18:33] Recompiled CSS in amp-access-scroll (671 ms)
[11:18:33] Recompiled CSS in amp-ad (670 ms)
[11:18:33] Recompiled CSS in amp-apester-media (670 ms)
[11:18:33] Recompiled CSS in amp-delight-player (667 ms)
[11:18:33] Recompiled CSS in amp-byside-content (667 ms)
[11:18:33] Recompiled CSS in amp-consent (663 ms)
[11:18:33] Recompiled CSS in amp-fit-text (661 ms)
[11:18:33] Recompiled CSS in amp-form (661 ms)
[11:18:33] Recompiled CSS in amp-fx-flying-carpet (660 ms)
[11:18:33] Recompiled CSS in amp-gwd-animation (659 ms)
[11:18:33] Recompiled CSS in amp-image-lightbox (658 ms)
[11:18:33] Recompiled CSS in amp-instagram (655 ms)
[11:18:33] Recompiled CSS in amp-live-list (651 ms)
[11:18:33] Recompiled CSS in amp-mathml (652 ms)
[11:18:33] Recompiled CSS in amp-next-page (651 ms)
[11:18:33] Recompiled CSS in amp-script (644 ms)
[11:18:33] Recompiled CSS in amp-sidebar (642 ms)
[11:18:33] Recompiled CSS in amp-story-auto-ads (604 ms)
[11:18:33] Recompiled CSS in amp-selector (603 ms)
[11:18:33] Recompiled CSS in amp-web-push (603 ms)
[11:18:33] Recompiled CSS in amp-image-viewer (601 ms)
[11:18:33] Recompiled CSS in amp-recaptcha-input (598 ms)
[11:18:33] Recompiled CSS in amp-twitter (597 ms)
[11:18:33] Recompiled CSS in amp-user-notification (597 ms)
[11:18:33] Recompiled CSS in amp-viz-vega (596 ms)
[11:18:33] Recompiled CSS in amp-access-laterpay (677 ms)
[11:18:33] Recompiled CSS in amp-access-laterpay (677 ms)
[11:18:33] Recompiled CSS in amp-playbuzz (655 ms)
[11:18:33] Recompiled CSS in amp-app-banner (686 ms)
[11:18:33] Recompiled CSS in amp-carousel (683 ms)
[11:18:33] Recompiled CSS in amp-image-slider (674 ms)
[11:18:33] Recompiled CSS in amp-lightbox (672 ms)
[11:18:33] Recompiled CSS in amp-list (670 ms)
[11:18:33] Recompiled CSS in amp-pinterest (667 ms)
[11:18:33] Recompiled CSS in amp-sticky-ad (658 ms)
[11:18:33] Recompiled CSS in amp-pan-zoom (616 ms)
[11:18:33] Recompiled CSS in amp-video-docking (613 ms)
[11:18:33] Recompiled CSS in amp-subscriptions (618 ms)
[11:18:33] Recompiled CSS in amp-lightbox-gallery (673 ms)
[11:18:33] Recompiled CSS in amp-base-carousel (725 ms)
[11:18:33] Recompiled CSS in amp-social-share (695 ms)
[11:18:33] Recompiled CSS in amp-date-picker (787 ms)
[11:18:33] Recompiled CSS in amp-subscriptions-google (852 ms)
[11:18:34] Recompiled CSS in amp-story (1.306 s)
[11:18:34] Recompiled CSS in amp-story (1.365 s)
[11:18:34] Processed 3p/recaptcha.max.html (016 ms)
[11:18:34] Processed 3p/nameframe.max.html (016 ms)
[11:18:34] Processed 3p/frame.max.html (016 ms)
[11:18:34] Transformed ampcontext-lib.js (060 ms)
[11:18:34] Transformed integration.js (067 ms)
[11:18:34] Transformed polyfills.js (169 ms)
[11:18:34] Transformed amp-shadow.js (041 ms)
[11:18:34] Transformed iframe-transport-client-lib.js (058 ms)
[11:18:34] Transformed video-iframe-integration.js (037 ms)
[11:18:34] Transformed amp.js (050 ms)
[11:18:34] Transformed recaptcha.js (055 ms)
[11:18:35] Transformed install-alp.js (978 ms)
[11:18:35] Transformed inabox-host.js (917 ms)
[11:18:35] Transformed amp-inabox.js (922 ms)
[11:18:35] Transformed web-worker.js (969 ms)
[11:18:35] Transformed examiner.js (1.102 s)
[11:18:35] Transformed amp-viewer-host.js (1.177 s)
[11:18:35] Compiled examiner.max.js (1.372 s)
[11:18:50] Compiled integration.js (16.352 s)
[11:18:50] Compiled video-iframe-integration.js (16.327 s)
[11:18:51] Compiled recaptcha.js (16.877 s)
[11:18:51] Compiled iframe-transport-client-lib.js (17.201 s)
[11:18:51] Compiled ampcontext-lib.js (17.205 s)
[11:18:51] Compiled ww.max.js (17.221 s)
[11:18:52] Compiled polyfills.js (17.826 s)
[11:18:52] Compiled amp-viewer-host.max.js (17.802 s)
[11:18:58] Compiled amp-shadow.js (23.803 s)
[11:18:58] Compiled amp.js (23.812 s)
[11:18:58] Compiled amp-inabox.js (23.799 s)
[11:18:58] No configs found in dist.3p/current/integration.js
[11:18:59] Compiled alp.max.js (24.907 s)
[11:18:59] Compiled amp-inabox-host.js (24.847 s)
[11:18:59] Enabled local development mode in dist.3p/current/integration.js
[11:18:59] Wrote prod AMP config to dist.3p/current/integration.js
[11:18:59] No configs found in dist/amp.js
[11:18:59] No configs found in dist/amp-shadow.js
[11:18:59] No configs found in dist/amp-inabox.js
[11:18:59] Enabled local development mode in dist/amp.js
[11:18:59] Enabled local development mode in dist/amp-shadow.js
[11:18:59] Wrote prod AMP config to dist/amp.js
[11:18:59] Enabled local development mode in dist/amp-inabox.js
[11:18:59] Wrote prod AMP config to dist/amp-shadow.js
[11:18:59] Wrote prod AMP config to dist/amp-inabox.js
[11:18:59] Finished 'build' after 28 s
[11:18:59] Starting 'test'...
[11:18:59] Run gulp help to see a list of all test flags.
[11:18:59] ⤷ Use --nohelp to silence these messages.
[11:18:59] ⤷ Use --headless to run tests in a headless Chrome window.
[11:18:59] Running tests against unminified code.
[11:18:59] --safari: Running tests on Safari.
[11:18:59] --files: Running tests in the file(s): test/unit/test-performance.js
[11:18:59] Webserver started at http://localhost:8081
[11:18:59] Started test responses server on localhost:8081
[11:18:59] Running test. Press Ctrl + C to cancel...
[11:19:00] Running tests locally...
START:
[11:19:01] Transforming tests with browserify...
.......................................................................................................................................................
[11:19:10] Done. Running tests...
performance
● should wait for visible resources
when viewer is not ready
● should queue up tick events
● should map tickDelta to tick
● should map tickDelta to non-zero tick
● should have max 50 queued events
● should add default optional relative start time on the queued tick event
● should drop events in the head of the queue
● should wait for visible resources
when viewer is ready,
config
● should configure correctly when viewer is embedded and supports csi
● should queue up tick events
● should configure correctly when viewer is embedded and does NOT support csi
● should map tickDelta to tick
● should configure correctly when viewer is embedded and does NOT support csi
● should configure correctly when viewer is not embedded
● should map tickDelta to non-zero tick
channel established
● should flush events when channel is ready
● should have max 50 queued events
channel not established
● should not flush anything
tickSinceVisible
● should always be zero before viewer is set
● should add default optional relative start time on the queued tick event
● should always be zero before visible
● should drop events in the head of the queue
● should calculate after visible
● should configure correctly when viewer is embedded and supports csi
● should be zero after visible but for earlier event
● should configure correctly when viewer is embedded and does NOT support csi
and performanceTracking is off
● should not forward queued ticks
● should configure correctly when viewer is embedded and does NOT support csi
● should ignore all calls to tick
● should configure correctly when viewer is not embedded
● should ignore all calls to flush
and performanceTracking is on
● should forward all queued tick events
● should flush events when channel is ready
● should have no more queued tick events after flush
● should not flush anything
● should forward tick events
● should always be zero before viewer is set
● should call the flush callback
● should always be zero before visible
coreServicesAvailable
document started in prerender
● should call prerenderComplete on viewer
● should call prerenderComplete on viewer even if csi is off
● should calculate after visible
● should be zero after visible but for earlier event
● should tick `pc` with delta=400 when user request document to be visible before before first viewport completion
● should tick `pc` with `delta=0` when viewport is complete before user request document to be visible
● should not forward queued ticks
document did not start in prerender
● should call prerenderComplete on viewer
● should ignore all calls to tick
● should tick `pc` with `opt_value=undefined` when user requests document to be visible
● should ignore all calls to flush
performance with experiment
● rtvVersion experiment
● should forward all queued tick events
● addEnabledExperiment should work
● should have no more queued tick events after flush
PeformanceObserver metrics
should forward paint metrics for performance entries
● created after performance service registered
● should forward tick events
should forward first input metrics for performance entries
● created after performance service registered
[11:19:11] Safari 12.0.3 (Mac OS X 10.13.6): Executed 34 of 34 (Skipped 0) 2 FAILED
● should call the flush callback
● should call prerenderComplete on viewer
● should call prerenderComplete on viewer even if csi is off
● should tick `pc` with delta=400 when user request document to be visible before before first viewport completion
● should tick `pc` with `delta=0` when viewport is complete before user request document to be visible
● should call prerenderComplete on viewer
● should tick `pc` with `opt_value=undefined` when user requests document to be visible
● rtvVersion experiment
● addEnabledExperiment should work
● created after performance service registered
● created after performance service registered
[11:19:11] Safari 12.0.3 (Mac OS X 10.13.6): Executed 34 of 34 (Skipped 0) SUCCESS
Finished in 1.605 secs / 0.413 secs @ 11:19:11 GMT-0500 (Eastern Standard Time)
SUMMARY:
● 66 tests completed
● 2 tests failed
FAILED TESTS:
performance
when viewer is ready,
channel established
● should flush events when channel is ready
Safari 12.0.3 (Mac OS X 10.13.6)
expected flush to have been called exactly 4 times, but it was called thrice
flush() at unknown
flush() at unknown
AssertionError@node_modules/chai/chai.js:9449:22
assert@node_modules/chai/chai.js:239:31
node_modules/sinon-chai/lib/sinon-chai.js:92:24
methodWrapper@node_modules/chai/chai.js:7824:30
[native code]
/Users/ericnyc/git/amphtml/test/unit/test-performance.js:231:45 <- /var/folders/pc/rxqr52ln52d95zdt4955q4qc00kz84/T/3708101bfe835f0f5341b283a9b357c8.browserify.js:72566:45
promiseReactionJob@[native code]
channel not established
● should not flush anything
Safari 12.0.3 (Mac OS X 10.13.6)
expected flush to have been called exactly thrice, but it was called twice
flush() at unknown
AssertionError@node_modules/chai/chai.js:9449:22
assert@node_modules/chai/chai.js:239:31
node_modules/sinon-chai/lib/sinon-chai.js:92:24
methodWrapper@node_modules/chai/chai.js:7824:30
[native code]
/Users/ericnyc/git/amphtml/test/unit/test-performance.js:259:45 <- /var/folders/pc/rxqr52ln52d95zdt4955q4qc00kz84/T/3708101bfe835f0f5341b283a9b357c8.browserify.js:72587:45
promiseReactionJob@[native code]
[11:19:11] Shutting down test responses server on localhost:8081
[11:19:11] ERROR: Karma test failed with exit code 1
[11:19:11] Finished 'test' after 12 s