##Overview
For Identity, we have a native implementation of persona for b2g. We have a quantity of native firefox (gecko) code to test, and we have a quantity of front-end (gaia) code to test. Our code is exercised through DOM API calls. To have assurance that all our code works correctly in concert, we require a robust suite of front-end tests. So we want to contribute to automated testing on b2g. This gist describes how someone writing gecko, gaia, apps, or all of the above can do that.
The following movie demonstrates a marionette test suite driving a gaia test_app to exercise our native gecko code:
I'll go through the steps from the point of view of someone who has to touch gaia, gecko, and apps code. Please try to ignore the sections that don't apply to you.
For b2g, writing front-end tests to exercise native code is a bit complicated.
You are working with three separate repositories:
- gecko (e.g., mozilla-central)
- gaia (all interface components, maybe also your test app)
- gaia-ui-tests (home of ATeam's marionette tests)
With code spread out like this, there's an inevitable chicken-and-egg problem with code updates and testing: Code has to land in mozilla-central and gaia before it can be tested; tests will fail until code lands. And there's no obvious way to create a patch that applies to gaia and gecko and test it before you submit. As a result, if someone files a bug against my code, the first order of business will be to determine whose commit (mine or someone else's) is actually at fault, and what repo that commit is in. Misdiagnosed bugs are not uncommon at this point, and that's a drag; but I think it's inevitable until automated testing has much greater coverage.
Gecko coders, notes that mochitests don't work for b2g. You will need to replace your mochitests with marionette tests.
Marionette is new. Kinks are still being worked out, and overall I have found that knowledge of its workings and interaction with gaia/gecko is unevenly distributed. That will no doubt change as more and more developers learn how to use marionette, and marionette continues to improve, but the current state of affairs is challenging for developers. But coming in as an engineer with no prior marionette experience, it can be hard to know where to go for help.
Hence this gist.
This is what you do as of December, 2012:
##1. Configure gecko
Do this if you are a gecko hacker. Other folks should be able to use a latest build.
If you are building your own mozilla-central or other gecko build, ensure that you are building with the following two flags:
ENABLE_MARIONETTE=1
ac_add_options --enable-tests
The latest builds ending in eng (which you can get here) should be built with these settings, but not all builds work. If in doubt, check with the #ateam channel in IRC.
##2. Configure gaia
It's possible to have marionette install apps to test, but as far as I know, you cannot push app code into the phone; you have to legitimately install it from a server somewhere. This feels risky to me, so I have opted to make a test_app that will appear on debug builds only. If you think you are going to install your test apps, skip this section. Otherwise, read on.
Build gaia like so:
rm -rf profile && DEBUG=1 make
Do not run make tests. These are obsolete. If the make target has not been removed, do not be deceived by the shiny make tests. It will only make sad.
If you are running on a desktop build (not device or emulator), you must add the following line to your profile/user.js:
pref("marionette.force-local", true); // <-- only for desktop environment
##3. Run the gaiamobile server
Everybody who runs a desktop build in debug mode needs to do this. Without this step, gaia will not work.
Install the b2g-scripts like so:
npm install b2g-scripts -g
These are maintained by @lightsofapollo, whom you may find in the #gaia channel.
Run the server:
b2g-scripts server -d test_apps -d apps --gaia $GAIA_PATH --port 8080
This is necessary for gaia built with DEBUG=1. Note the -d options there.
##4. Launch b2g
You should now be able to run b2g with your gaia profile. On the mac, you have to run the .app file; on Linux, just run b2g. On Mac:
$B2G_BUILD/dist/B2GDebug.app/Contents/MacOS/b2g \
-profile $GAIA_PATH/profile \
-jsconsole
(The jsconsole is optional; I normally do not run it. But sometimes it gives output that you won't get in the console, such as CSP debugging info or front-end js errors.)
If all goes well, your starup messages will include lines like:
1355259164517 Marionette INFO MarionetteComponent loaded
1355259164519 Marionette INFO marionette enabled, loadearly: true
1355259164638 Marionette INFO marionette initializing at final-ui-startup
1355259164715 Marionette INFO marionette-actors.js loaded
1355259164718 Marionette INFO marionette listener opened
If you get any warnings about "Firefox cannot find gaiamobile", check that you are
running the b2g-script server.
##5. Run a Marionette test
Clone the gaia-ui-tests
cd gaia-ui-tests
Run a test:
gaiatest gaiatest/tests/test_clock.py --address localhost:2828
You should see things happening in the b2g ui. Yay.
(With the clock test, I presently get one pass, one fail.)
Did it crash with "nodename nor servname provided, or not known"? And you're on a Mac?
Edit mozhttpd/iface.py and hard-code it to return "127.0.0.1".
##6. Write a Marionette test
There is a helpful guide for running marionette from the python shell.
Here is the beginnings of a test_apps/identity test:
https://github.com/jedp/gaia/compare/identity-test-app
Follow the steps above to build gaia and run the b2g-scripts server.
Here is marionette code to run an app and click a button in it. This works from the python shell:
from marionette import Marionette
m = Marionette('localhost', 2828)
m.start_session()
m.import_script('gaiatest/atoms/gaia_lock_screen.js')
m.import_script('gaiatest/atoms/gaia_apps.js')
m.execute_async_script('GaiaLockScreen.unlock()')
app = m.execute_async_script("GaiaApps.launchWithName('IdentityTest')")
m.switch_to_frame(app['frame'])
m.marionette.wait_for_element_present('css selector', 'ready')
m.find_element('id', 'request').click()Look at that! It opened the Trusty UI!
What? It didn't for you? I know. This is because the synthetic gaia click events are not low-level enough for the dom to think they're actual user interaction. So this line bounces the marionette click:
http://hg.mozilla.org/mozilla-central/file/b11065872128/dom/identity/nsDOMIdentity.js#l118
I've filed a bug for this. In the mean time, I've patched nsDOMIdentity to lift
the requirement for native events if this pref is set: dom.identity.syntheticEventsOk.
The marionette suite sets this variable in the setUp method.
You set prefs in marionette using SpecialPowers:
marionette.execute_script(
"""SpecialPowers.setBoolPref("%s", true);""" % NAME_OF_PREF_TO_SET,
special_powers=True);
Note the special_powers=True. To get the value of the pref:
marionette.execute_script(
"""return SpecialPowers.getBoolPref("%s")""" % NAME_OF_PREF_TO_GET)
To make the movie above, I used:
- Get the test app landed in gaia
- Get the tests landed in gaia-ui-tests
You write:
rm -rf profile && DEBUG=1 make
that removes profile, and generates profile-debug
in step 4:
-profile $GAIA_PATH/profile
should be
-profile $GAIA_PATH/profile-debug