Partial Selenium Grid integration support with tools like AutoIt, Sikuli, etc.

PartialSeleniumGridIntegrationWithAutoItExample.java

import static org.junit.Assert.*;

import org.junit.After;
import org.junit.Before;
import org.junit.Test;

import org.openqa.selenium.*;
import org.openqa.selenium.remote.*;
import java.net.URL;

public class DemoTest {
	
	public WebDriver driver;
	public DesiredCapabilities capabilities;

	@Before
	public void setUp() throws Exception {
		capabilities = DesiredCapabilities.firefox();
		driver = new RemoteWebDriver(new URL("http://localhost:4444/wd/hub" ), capabilities);
	}

	@After
	public void tearDown() throws Exception {
		driver.quit();
	}

	@Test
	public void test() throws Exception {
		// Use RemoteWebDriver, grab actual node host info
		driver.get("http://www.google.com");		
		String sessionId = ((RemoteWebDriver) driver).getSessionId().toString();
		//grid info extractor from: https://gist.github.com/krmahadevan/1766772
		String nodeHost = GridInfoExtracter.getHostNameAndPort("localhost", 4444, sessionId)[0];		
		System.out.println("Extracted hostname: "+nodeHost);
		
		// Now use node host info to handle running AutoIt on that specific node, assuming all needed files deployed to all nodes
		
		// Case 1 - PSExec.exe from Windows host (that is executing this Selenium code) to Selenium node that is Windows host
		//String psexecCmd = "C:\\LocalMachinePathTo\\psexec.exe \\\\%s -u %s -p %s -i C:\\GridNodeMachinePathTo\\autoitCompiledScript.exe";
		//Process p = Runtime.getRuntime().exec(String.format(psexecCmd,nodeHost,"jdoe","hisPassword"));
		//p.waitFor();
		
		// Case 2 - winexe from *nix host (that is executing this Selenium code) to Selenium node that is Windows host
		// Example reference: http://secpod.org/blog/?p=661
		//String winexeCmd = "/LocalMachinePathTo/winexe -U domainName/jdoe%hisPassword //"+nodeHost+" C:\\GridNodeMachinePathTo\\autoitCompiledScript.exe";
		//Process p = Runtime.getRuntime().exec(winexeCmd);
		//p.waitFor();
		
		// Case 3 - have SSH installed on Windows-based Selenium nodes, now just connect to node host via SSH to run a command, i.e. execute AutoIt script binary
		// no sample code given for Java, but maybe this gives you ideas on Java SSH connection:
		// http://stackoverflow.com/questions/995944/ssh-library-for-java

		// Case 4 - if you have implemented a custom web service (XML-RPC/SOAP/REST) for AutoIt that listens for requests/commands
		// just connect to it via (Java) HTTP library to "http://"+nodeHost+"/someWebServicePath/someCommand" (via GET or POST)
		// details not covered, this is just an example. Most people won't be taking this route due to customization & complexity in
		// building the web service first
		
		// Case 5 - using AutoItDriverServer that is also running on Selenium nodes
		//DesiredCapabilities autoitCapabilities = new DesiredCapabilities();
		//autoitCapabilities.setCapability("browserName", "AutoIt");
		//WebDriver autoitDriver = new RemoteWebDriver(new URL("http://"+nodeHost+":4723/wd/hub"), autoitCapabilities);
		//autoitDriver.findElement(By.id("133")).click();
		//and whatever other AutoItX commands to call that you normally have in the AutoIt script that you compile into binary
		//for more ideas on AutoIt commands issued as WebDriver commands with respect to Selenium integration, see:
		//https://github.com/daluu/AutoItDriverServer/blob/master/sample-code/SeleniumIntegrationWithAutoItDriver.py
		
		//or if you are old school, and want to do that same approach even with AutoItDriverServer,
		// assuming you have first set AutoItScriptExecuteScriptAsCompiledBinary to True in autoit_options.cfg file before starting AutoItDriverServer:
		//((JavascriptExecutor) autoitDriver).executeScript("C:\\GridNodeMachinePathTo\\autoitCompiledScript.exe");
		
		// Case for Sikuli integration, using https://github.com/enix12enix/sikuli-remote-control
		//RemoteScreen rs = new RemoteScreen(nodeHost);
		//rs.setMinSimilarity(0.9);
		//rs.click("D://test.png");
	}

}

Note that the nature of such integration won't work for SauceLabs and similar services unless they expose ability to run custom tools atop their virtual browser sessions. This hack is meant for internal grid deployments.

Hi Daluu

Can we have the AutoITdriverServer in standalone executable so that we can use it as InternetExplorerDriver.
I tried using py2exe but its failing because of some unknown descripencies.

Thanks and Regards
Kishor Srivastava

Kishor, maybe when I get around to making a .NET port of the server.

Hello Daluu,
I was trying this approach and to execute the selenium script and AutoIt script in server machine.
Till extract hostname I'm getting hostname like : 10.90.67.190

And after that I have modified the case 5 like :
DesiredCapabilities autoitCapabilities = new DesiredCapabilities();
autoitCapabilities.setCapability("browserName", "AutoIt");
WebDriver autoitDriver = new RemoteWebDriver(new URL("http://"+nodeHost+":4723/wd/hub"), autoitCapabilities);
System.out.println("Open Calculator");
autoitDriver.get("calc.exe");
Thread.sleep(4000);
autoitDriver.switchTo().window("Calculator");
Thread.sleep(1000);
System.out.println("Start Operation");
autoitDriver.findElement(By.id("139")).click(); // 3
Thread.sleep(1000);

It is working fine till printing Start Operation but after that it is failing and i'm getting error like :

Exception in thread "main" org.openqa.selenium.WebDriverException: AutoIt failed to click element [ID:139].
Command duration or timeout: 20 milliseconds

Please help me to resolve this issue. I need this urgently.

Thanks and Regards,
Ashish

Hi @ashish9308, did you ever solve your issue? Sorry I didn't notice the comment so it's been so long. That issue sounds like it can't find the calculator's 3 button, intended to be ID 139. Best to confirm you can detect that element with AutoIt's spy/inspector tool, I forget its name, or using some other Windows GUI inspector tool. And you have to be sure the Calculator window is visible and ideally the top most active window, for the automation to run successfully.

The calculator element IDs may have changed between versions of Windows. I haven't done anything with this in a long time.