Angular E2E Tests (Protractor) - Performance Measurement

README.md

Protractor and Performance Test

Protractor is the end to end test case runner for AngularJS. These end to end test cases can be repurposed to record performance metrics when the scenario is being run. This is a sample repository with an example of how this can be done.

Usage

Step 0 - Setup

Install all dependencies using npm install

Step 1 - Prepare configuration

Protractor currently supports only synchronous configurations. Here is the issue asking for support of asynchronous configurations. Till this feature is added, you would need to generate a temporary configuration that would have the browser parameters required for performance analysis. Run to following command to generate conf.js.tmp.

node index.js --config conf.js

Step 2 - Run Protractor Tests

Once the configration file is generated, run protractor against this using

node_modules/.bin/protractor conf.js.tmp

This will run the example spec and also record the performance metrics. Note the calls to perfRunner in lines 4 8 and 16 that initialize, start and stop the performance metrics.

Using it in your projects

To start using it in your projects,

1. Add browser-perf to your project
2. Copy index.js and use it in your test case to mark the start and stop of the scenario.
3. Ensure that in your build, you generate the configuration, live done in Step 1 above.

conf.js

// An example configuration file.
exports.config = {
  // The address of a running selenium server.
  seleniumAddress: 'http://localhost:4444/wd/hub',

  // Capabilities to be passed to the webdriver instance.
  capabilities: {
    'browserName': 'chrome'
  },

  // Spec patterns are relative to the current working directly when
  // protractor is called.
  specs: ['example.spec.js'],

  // Options to be passed to Jasmine-node.
  jasmineNodeOpts: {
    showColors: true,
    defaultTimeoutInterval: 300000
  },

  allScriptsTimeout: 200000
};

example.spec.js

var PerfRunner = require('./index.js');
describe('angularjs homepage', function() {

	var perfRunner = new PerfRunner(browser.params.perf);

	it('should greet the named user', function(done) {

		perfRunner.start(); // Start monitoring before the scenario starts running

		browser.get('http://www.angularjs.org');
		element(by.model('yourName')).sendKeys('Julie');
		var greeting = element(by.binding('yourName'));
		expect(greeting.getText()).toEqual('Hello Julie!');

		// Use this once the scenario is over to get the results
		perfRunner.stop().then(function(data) {
			console.log(data);
		});

	});
});

index.js

#!/usr/bin/env node

var browserPerf = require('browser-perf');

// Command line usage to geneate tmp config file
if (process.argv.length >= 2 && process.argv[2] === '--config') {
	var fs = require('fs'),
		path = require('path');

	var configFile = process.argv[3];
	if (!configFile) {
		throw 'Specify Protractor Configuration file';
	}
	var cfg = require('./' + configFile).config;

	var runner = new browserPerf.runner({
		selenium: cfg.seleniumAddress,
		browsers: [cfg.capabilities]
	});

	runner.config(function(err, data) {
		cfg.capabilities = data.browsers[0];
		cfg.params = cfg.params || {};
		cfg.params.perf = data;
		console.log('Configuration written to', path.basename(configFile) + '.tmp', 'Run protractor with this new file now');
		require('fs').writeFileSync(path.basename(configFile) + '.tmp', 'exports.config=' + JSON.stringify(cfg));
	});
}

// Exported code used inside the test cases

var PerfRunner = function(cfg) {
	this.browserPerfRunner = new browserPerf.runner(cfg);
	this.sessionId = null;
}

PerfRunner.prototype.start = function() {
	var me = this,
		flow = protractor.promise.controlFlow();

	return flow.execute(function() {
		var d = protractor.promise.defer();
		(function() {
			if (me.sessionId === null) {
				return browser.getSession().then(function(session) {
					return protractor.promise.fulfilled(session.getId());
				});
			} else {
				return protractor.promise.fulfilled(me.sessionId);
			}
		}()).then(function(sessionId) {
			me.sessionId = sessionId;
			me.browserPerfRunner.start(me.sessionId, function(err, data) {
				if (err) {
					d.reject();
				} else {
					d.fulfill();
				}
			});
		});
		return d.promise
	});
};

PerfRunner.prototype.stop = function() {
	var me = this,
		flow = protractor.promise.controlFlow();
	return flow.execute(function() {
		var d = protractor.promise.defer(),
			flow = protractor.promise.controlFlow();
		me.browserPerfRunner.stop(function(err, data) {
			if (data) {
				d.fulfill(data);
			} else {
				d.reject(err);
			}
		});
		return d.promise;
	});
}

module.exports = PerfRunner;

package.json

{
  "name": "protractor-perf",
  "version": "0.0.1",
  "description": "Reusing Angular e2e tests for performance analysis",
  "main": "index.js",
  "dependencies": {
    "browser-perf": "~0.1.3",
    "protractor": "~0.19.0"
  },
  "devDependencies": {},
  "scripts": {
    "test": "node index.js"
  },
  "keywords": [
    "perf",
    "web",
    "webperformance",
    "performance",
    "angular",
    "protractor",
    "browser-perf"
  ],
  "author": "Parashuram <code@r.nparashuram.com>",
  "license": "Apache 2.0"
}

Hi,

I've just tested this setup in one of my company project and it seems to work well !
Good job, can't wait to test it more !

One thing I had to do is copy/paste the code from onPrepare protractor config to the generated conf file, because it was not copied during the config generation

Thanks for sharing this!

I've been trying to integrate this with a project I'm working on, and everything works great apart from one little thing, which I'm hoping I can get some advice on:

Using directConnect in Protractor, when Chrome opens the about:tracing tab, the test suite gets stuck, and will only continue if I manually go back to the original tab.

This is reproducible with the original code from this gist, but with directConnect : true set in conf.js.

It seems like it's getting stuck on me.browserPerfRunner.stop(function(err, data) {. I can get the previous line to log something, but anything inside this stop() method won't log, and it looks like the function isn't executing at all. I'm not sure if this is because a previous promise is timing out; I'm still trying to understand how all these synchronous calls relate to each other.

I suspect Selenium is losing control of the original tab, and I've tried switching back to the original tab using browser.switchTo(). This doesn't seem to change the visible tab, but some logging in protractor suggests that protractor thinks it's successfully switched. Unfortunately this isn't enough to get the tests to resume automatically.

Thanks in advance if you can think of anything, I'm lost.

Hi ,
I am observing '[11:32:38] E/launcher - Error: Error: Cannot find module './index.js'' when I try to execute the configuration file. I tried placing index file in different location and still observed the same error

This is my file structure. Not sure what is wrong with it. Please help me

Thanks for sharing this Parashuram!!! It works like a charm!!!

premkh9, in your case, considering the folder structure displayed in the stack trace, the relative path of index.js mentioned in the spec is wrong. It will work if you change that line to
var PerfRunner = require('../../../../../../../../index.js') if the index.js file is in the project root.

OR

Just simply place that index.js file in the same directory containing the spec file