Using robotframework with Plone

login.txt

*** Settings ***

Documentation  A test suite with a single test for valid login. This test has
...            a workflow that is created using keywords from the resource file.
Resource       resource.txt


*** Test Cases ***

Valid Login
    ${username} =  Get site owner name
    ${password} =  Get site owner password
    Open Browser To Login Page
    Input Username  ${username}
    Input Password  ${password}
    Submit Credentials
    Welcome Message is Being Shown
    [Teardown]  Close Browser

resource.txt

*** Settings ***

Documentation  A resource file containing the application specific keywords
...            that create our own domain specific language. This resource
...            implements keywords for testing HTML version of the test
...            application.
Library        Selenium2Library
Library        plone.app.theming.testing.Keywords


*** Variables ***

${SERVER}        localhost:55001
${BROWSER}       firefox
${DELAY}         0
${SITE}          plone
${LOGIN URL}     http://${SERVER}/${SITE}/login
${WELCOME URL}   http://${SERVER}/${SITE}/front-page


*** Keywords ***

Open Browser To Login Page
    Open Browser  ${LOGIN URL}  ${BROWSER}
    Maximize Browser Window
    Set Selenium Speed  ${DELAY}
    Page should contain element  css=#login-form

Go To Login Page
    Go To  ${LOGIN URL}
    Page should contain element  css=#login-form

Input Username  [Arguments]  ${username}
    Input Text  __ac_name  ${username}

Input Password  [Arguments]  ${password}
    Input Text  __ac_password  ${password}

Submit Credentials
    Click Button  Log in

Welcome Message is Being Shown
    Page should contain  You are now logged in

Login Should Have Failed
    Location Should Be  ${LOGIN URL}
    Title Should Be  Error Page

robots.cfg

#
# Step 2 - Set versions (and use some packages from source for now)
#

[buildout]
extends = buildout.cfg

[versions]
robotsuite = 0.5.0
robotframework = 2.7.3
robotframework-selenium2library = 1.0.1
docutils = 0.8.1
decorator = 3.3.3
selenium = 2.25.0

[omelette]
eggs += ${test:eggs}

setup.py

#
# Step 1 - Declare dependencies (setup.py)
#

      extras_require={
        'test': [
            'plone.app.testing',
            'robotsuite',
            'selenium', # XXX: This is here because of a missing dependency declaration in robotframework-selenium2library 1.0.1
            'decorator', # XXX: This is here because of a missing dependency declaration in robotframework-selenium2library 1.0.1
            'robotframework-selenium2library'
        ],
      },

test_robot.py

#
# Step 4 - Create a test suite using robotsuite
#


import unittest2 as unittest

from plone.testing import layered
from robotsuite import RobotTestSuite

from plone.app.theming.testing import THEMING_ACCEPTANCE_TESTING


def test_suite():
    return unittest.TestSuite([
        layered(RobotTestSuite('acceptance-tests'),
            layer=THEMING_ACCEPTANCE_TESTING),
    ])

testing.py

from plone.testing import z2
from plone.app.testing import PloneSandboxLayer
from plone.app.testing import PLONE_FIXTURE
from plone.app.testing import applyProfile

from zope.configuration import xmlconfig

from plone.app.testing.layers import IntegrationTesting
from plone.app.testing.layers import FunctionalTesting


class Theming(PloneSandboxLayer):
    defaultBases = (PLONE_FIXTURE,)

    def setUpZope(self, app, configurationContext):
        # load ZCML
        import plone.app.theming.tests
        xmlconfig.file('configure.zcml', plone.app.theming.tests, context=configurationContext)

        # Run the startup hook
        from plone.app.theming.plugins.hooks import onStartup
        onStartup(None)

    def setUpPloneSite(self, portal):
        # install into the Plone site
        applyProfile(portal, 'plone.app.theming:default')


class Keywords(object):
    """Robot Framework keyword library
    """

    def get_site_owner_name(self):
        import plone.app.testing
        return plone.app.testing.interfaces.SITE_OWNER_NAME

    def get_site_owner_password(self):
        import plone.app.testing
        return plone.app.testing.interfaces.SITE_OWNER_PASSWORD

THEMING_FIXTURE = Theming()
THEMING_INTEGRATION_TESTING = IntegrationTesting(bases=(THEMING_FIXTURE,), name="Theming:Integration")
THEMING_FUNCTIONAL_TESTING = FunctionalTesting(bases=(THEMING_FIXTURE,), name="Theming:Functional")
THEMING_ACCEPTANCE_TESTING = FunctionalTesting(bases=(THEMING_FIXTURE, z2.ZSERVER_FIXTURE,), name="Theming:Acceptance")

Note - the test example fails - it's based on the demo app from the Robotframework Selenium2Library. It's just something to see some tests being run. Test output goes in parts/test. Tests are run with the usual bin/test. Note that to filter with -t, use "bin/test -s -t robotsuite" as the test cases are all called "robotsuite.".

I'll try this out (and should be able to complete it) later today... At least, you shouldn't use SELENIUM_FIXTURE, because robotframework-selenium2library takes care of the Selenium. You should use bare z2.ZSERVER_FIXTURE instead.

As @emanlove has reported, there are currently two approaches for isolated Plone-tests using RobotFramework:

1. robotsuite wraps RobotFramework-tests into unittest-testsuite (similarly how doctests are wrapped) and allows them to be run with zope.testrunner
2. @gotcha's plone.act re-implements parts of plone.app.testing as RobotFramework keywords to allow isolated (and layered) Plone-tests using pure RobotFramework testrunner

Note: in this updated version, login.txt and resource.txt live in an acceptance-tests subdirectory next to the test_acceptance.py boilerplate.

Ok. See https://gist.github.com/3331735 for successfully passing version. (I wonder, if there's a way to "merge" changes from cloned gists.)

A few notes:

• You don't need to require Selenium explicitly. Selenium2Library is enough.
• Unless you need drag and drop, you shouldn't need to use the development version of Selenium2Libarary. The latest release 1.0.1 work pretty well already.
• My clone has also an example of implementing your own Python keywords (e.g. for retrieving plone.app.testing defined usernames).
• Robotsuite generates per-test Robot Framework reports under buildout-directory/parts/tests. (Too bad, it still lacks good merged test report for all Robot Framework tests, because Robot Framework's own report merging tool doesn't work very well.)

Obviously, in the future, plone.app.testing could provide the base set of keywords to use with Robot Framework. @gotcha's plone.act is good reading: https://github.com/gotcha/plone.act/blob/master/src/plone/act/plone.txt

About robotsuite vs. plone.act (actually zope.testrunner vs. pybot-testrunner):

• I admit that using plone.act and pybot without zope.testrunner would be easier for people, who haven't learned zope.testrunner or plone.app.testing yet. Also, pybot has many features missing from zope.testrunner.
• Still, I've been working for robotsuite, because I'd like to run Robot Framework tests with old tests
  and benefit from the other zope.testrunner-compatible packages, e.g. CoreJet (see also: corejet.robot).

@datakurre: cool - I like your approach a lot. I didn't realise it was that easy to define custom keywords.

I think the robotsuite approach is the only one that makes sense, to be honest. Without it, we don't get proper test isolation or fixture management and tests need to use the UI to tear down state they set up. That's incredibly error-prone, and I've never seen it work well in other contexts.

If you look at the latest gist, we can still have a situation where you just dump files into a directory, so we can probably make this relatively easy to get started with. We could probably even create a buildout recipe that creates the relevant boilerplate. Something like the plone-devstart approach could also work, where you have a non-redistributable package for project-specific customisations.

corejet.robot is really cool. ;-)

Oh, I just realised why I used Selenium2Libarary from svn: there's an undeclared dependency on the 'decorator' module. It also doesn't pull in 'selenium' as far as I can tell. In fact, I think the released version isn't packaged with setuptools at all.

s/svn/github/ - old habits die hard ;)

I hope that @gotcha could comment about test isolation in plone.act. I guess, he's been using it without issues so far. I don't have enough low level experience with zope/plone.testing to review his isolation keywords.

I wonder, why I don't have similar issues with using the released version of selenium2library. It might be, because I've been running robot test only with Python 2.7.

A few more things to know about Robot Framework:

• Keywords are looked up case-insensitively.
• You can create BDD-style test clauses beginning with Given, When, Then or And. Those simple stripped before looking for the related keyword: http://robotframework.googlecode.com/hg/doc/userguide/RobotFrameworkUserGuide.html?r=2.7.2#behavior-driven-style
• You can use Cucumber-style embedded arguments in your custom keywords: http://robotframework.googlecode.com/hg/doc/userguide/RobotFrameworkUserGuide.html?r=2.7.2#behavior-driven-development-example

About:

Note that to filter with -t, use "bin/test -s -t robotsuite"

Would you have any objections against this behavior in the current master of robotsuite:

$ bin/test --list-tests
Listing my.app.testing.MyApp:Acceptance tests:
    Can_do_A (functional/activation.txt)
    Can_do_B (functional/activation.txt)
    Can_do_C (functional/payment.txt) #current

In summary, robotsuite.RobotTestCase is replaced with a relative path of the test (including RF-testsuite folder) and appended with Robot Framework tags.

Now you could filter tests with:

$ bin/test -t functional
$ bin/test -t activation.txt
$ bin/test -t \#current