These cases are based upon assumptions that I have of what "perfect" or really good functional tests should be like.
- Have a setup that cannot fail, or if it does fail, it is covered by a lower level test (integration/persistence).
- Test what it is supposed to test after, without too much overlap with same-layered tests to avoid dependency issues. (Fetch,Get integration tests often have this issue)
- Readable. Anyone should be able to look at a test and be able to understand what it does
- Maintainable. There should be a clean hierarchy of the macros that prevent tests from breaking each other, but also allow fixing certain macros to fix ones that it should depend on.
Pre-req Notes: Good test qualities
Stability(integraion example, unqiue failures), Speed(not taking too long / reducing load), Maintainability(being able to know how to fix things), Accuracy(failing when it should)
Test Layers
Persistence, Service, Front-end(*DisplayContext.java *.jsp, *.es.js, *.soy), Functional
API call things we don't care about, and then extensively test the functional aspects of things we care about.
Warning Assertions Stability
Another main issue with tests are randomly flaky failures. The success message is a common factor in randomly causing tests to fail. When a success message is expected to appear, we will assert the success message is present. The success message also has a chance to block certain locators if not closed, so we also attempt to close it if it is not yet closed. However, because there are chances that a success message may get accidently taken out, or even the issue where the success message automatically dissapears between when we assert that it is present, and that when we close it. Because of this, ideally we would want success messages to be a warning result when it fails, and not an error that ends the test.
Poshi VS Java: There does not seem to be any available way to get an element in Poshi without throwing the error, which leads to a main limitation with Poshi, we can't attempt to get a WebElement. If we fail to retrieve it, we can decide how we want to handle it in the test. But currently, even if we were to constantly create new functions, there is a lot more flexibility with being able to use Java for testing with Web Elements.
Poshi Solution:
- Request CI a function that returns if a locator is found or not without failing, and a manual warn command
- Wait for CI to implement, push upstream, and then re-release Poshi
macro viewSuccessMessage {
var successPresent = getPresent(Message#SUCCESS);
if ("${successPresent}" == "TRUE") {
warn("Success message not present.");
} else {
SuccessMessage.closeSuccessMessage(); //still might fail due to innate wait for SPARefresh, etc.
}
Java Solution:
public void viewSuccessMessage(String successMessageLocator) {
WebElement successMessage = getWebElement(successMessageLocator);
if (successMessage) {
closeSuccessMessageIfPossible(successMessage); //logic doesnt matter for now, this is an example
return;
}
LogUtil.warn("Success message not present"); //How we want to warn doesn't matter, this is an example
}So although this issue is achievable in both, in Java this would take around a couple of minutes to create, where as with Poshi, this would take at the minimum 1 week to solve one specific problem.
Remote API Calls Stability, Speed Developers will likely be willing to help with this
Remote API calls offer a HUGE improvement to the stability and performance of functional tests. Most API calls can save around 1 minute of time per API Call. With thousands of testcases, utilizing API Calls for all of setup could easily improve test speeds from 50-80%. However, the most beneficial aspect of API calls are test stability. With such high risks for locators to change, causing functional tests from being able to perform every action 100% of the time, reducing the amount of UI/Front-end setup would reduce the severity of Test Fixes losing a lot of scope for testing.
Performing the API Calls with Poshi or with Java should offer around the same benefit, however the main issue is with how hacky these API calls need to be when used with Poshi. Examples will be shown below
Service Context
Service Context is very important when making service calls to allow the user to be able to further customize the service when modifying an asset, such as creating an asset in draft mode, applying tags/categories, applying model permissions, or even setting which user created the asset. If we are creating an asset via API, at the minimum we need to be able to apply guestPermissions by default to all assets, otherwise when testing if a guest can view the asset, it will fail for our functional tests since they have not been applied.
Setting the serviceContext for Poshi, is quite frankly, a huge mess. First, we need to be able to create JSON Objects / JSON Arrays at the bare minimum. And with only strings available, it can get quite messy.
Poshi Example (Helper functions)
@summary = "Creates an empty JSON Array object if it does not yet exist in the JSON"
macro _addJSONArray {
if ((!(isSet(JSON))) || (!(isSet(key)))) {
fail("Error, 'JSON' and 'key' must all be set");
}
//check if object already exists
var bool = JSONUtil2._isJSONArrayPresent(
JSON = "${JSON}",
key = "${key}");
if ("${bool}" == "TRUE") {
echo("JSON array ${key}' already added, returning original JSON");
} else {
if ("${JSON}" == "{}") {
var JSON = '''{"${key}" : []}''';
} else {
var JSON = StringUtil.regexReplaceFirst("${JSON}","(.*)\}","$1, "${key}": []}");
}
}
return "${JSON}";
}
@summary = "Adds values to a JSON Array object, but fails if the value already exists"
macro _addJSONArrayValues {
if ((!(isSet(JSON))) || (!(isSet(key))) || (!(isSet(values)))) {
fail("Error, 'JSON', 'key', and 'values' must all be set");
}
var bool = JSONUtil2._isJSONArrayPresent(
JSON = "${JSON}",
key = "${key}");
if ("${bool}" == "FALSE") {
fail("Error, '${key}' does not exist in the JSON");
}
for (var value : list "${values}") {
var temp = RegexUtil.replace("${JSON}","\"${key}\"\s*:\s*(\[.*?\])","1");
if (contains(""${temp}"",""${value}"")) {
fail("'${value}' already exists inside 'JSON'");
}
if ("${temp}" == "[]") {
//if empty no need to add comma
var JSON = StringUtil.regexReplaceFirst("${JSON}","(\"${key}\"\s*:\s*)\[\]", "$1["${value}"]");
} else {
var JSON = StringUtil.regexReplaceFirst("${JSON}","(\"${key}\"\s*:\s*)(\[.*?)\]", "$1$2, "${value}"]");
}
}
return "${JSON}";
}
@summary = "Adds a new JSON object with a set value, but fails if the object already exists"
macro _addJSONObject {
if ((!(isSet(JSON))) || (!(isSet(key))) || (!(isSet(value)))) {
fail("Error, 'JSON', 'key', and 'value' must all be set");
}
var bool = JSONUtil2._isJSONObjectPresent(
JSON = "${JSON}",
key = "${key}");
if ("${bool}" == "TRUE") {
fail("Error, '${key}' already exists in the JSON");
}
if ("${JSON}" == "{}") {
var JSON = '''{"${key}" : ${value}}''';
} else {
var JSON = StringUtil.regexReplaceFirst("${JSON}","(.*)\}","$1, "${key}": ${value}}");
}
return "${JSON}";
}
@summary = "Checks if a JSON Array exists inside JSON"
macro _isJSONArrayPresent {
var temp = RegexUtil.replace("${JSON}","\"(${key})\"\s*:\s*\[.*?\]","1");
if ("${temp}" == "${key}") {
return "TRUE";
} else {
return "FALSE";
}
}
@summary = "Checks if a JSON Object exists inside JSON"
macro _isJSONObjectPresent {
var temp = RegexUtil.replace("${JSON}","\"(${key})\"\s*:\s*","1");
if ("${temp}" == "${key}") {
return "TRUE";
} else {
return "FALSE";
}
}Implementation: https://github.com/liferay/liferay-portal/blob/master/portal-web/test/functional/com/liferay/portalweb/macros/JSONServiceContextUtil.macro
if the jsonObject was available within Poshi, it would be much more simpler to implement. Although technically we can implement various JSON Object utils within a Poshi Util and have them be casted as a string after, we will have to go through the process of waiting for the CI team to implement them, making sure they work, and then waiting for the next Poshi release. It is actually less work to create our own Pseudo JSON Object using regex manipulation, in the case that we need to modify it in the future or create more functions.
Other object types
Because of the importance of remote API calls, many times we need to make calls using objects that are not of type string. Files, Maps, typeSettings, and even XML formatted Strings may be required. Many of these can be easily set in Java with a couple of lines and importing the required modules (if required), but in Poshi, regex needs to be used to create these data types in a hacky way.
Poshi Map Implementation:
@summary = "Helper function to convert a localized list into JSON"
macro _convertLocalizedListToMap {
// Create list using '${listMap'
var localizedMap = "{";
for (var i : list "${listMap}") {
var locale = StringUtil.extractFirst("${i}", ":");
var translated = StringUtil.extractLast("${i}", ":");
var localizedMap = '''${localizedMap}"${locale}":"${translated}",''';
}
// Replace last comma with }
var localizedMap = RegexUtil.replace("${localizedMap}", "(.*)(?=,)", "1");
var localizedMap = "${localizedMap}}";
return "${localizedMap}";
}Ticket for Map workaround: https://issues.liferay.com/browse/LRQA-49144 This ticket wouldn't even want to use the map workaround above, since it only wants to format the data into a map so it can be iterated through cleanly. So the above macro would not be enough for it, since it still needs to retrieve the map info, which would not be very easy to do as it requires a lot of hard regex. Many different workarounds may be created like this, creating many bandaids to re-implements part of a Map.
Poshi XML for WC Implementation
@summary = "Helper function to help input localized content html info"
macro _localizedConvertToXML {
// Get list of localizations used
var localeList = "";
for (var i : list "${contentMap}") {
var locale = StringUtil.extractFirst("${i}", ":");
var localeList = "${localeList},${locale}";
}
var localeList = RegexUtil.replace("${localeList}", ",(.*)", "1");
// Build dynamic content
var dynamicContent = "";
for (var i : list "${contentMap}") {
var locale = StringUtil.extractFirst("${i}", ":");
var translated = StringUtil.extractLast("${i}", ":");
var dynamicContent = '''${dynamicContent} <dynamic-content language-id="${locale}">${translated}</dynamic-content>''';
}
// Build XML
var contentXML = '''<root available-locales="${localeList}" default-locale="en_US"> <dynamic-element name="content" type="text_area" index-type="text"> ${dynamicContent} </dynamic-element> </root> ''';
return "${contentXML}";
}As you can see, there is a lot of heavy logic and re-inventing of the wheel required (and the reinvention is more like a minimum effort prototype) when using Poshi, instead of being able to use the object types naturally, which costs extra time and reduces maintainability due to the complexity of the workaround.
Maintainability With all of these API call workaround needed in order to perform them, their maintainability is very hard. If there ever comes a time where some of these helper functions need to be fixed, the difficulty of fixing them would actually be very, very, difficult, especially if the person that has to fix them is not well versed in regular expressions, as some of the workarounds are pretty complex, as they require both knowledge of how the structures of certain API calls require the objects to be like, as well as knowledge of regex manipulation (regex patterns need to be very clean as well to prevent incorect parsing).
So in general API calls in Poshi have a much higher skill cap than in Java (imagine trying to perform division with Assembly Language versus C). One is very simple, where as the other requires a bunch of complex logic.
Parameter Checking Maintainability, Stability
In Poshi, there are very few times where it makes sense to not declare a variable as "null" explicity when calling a macro. Imagine a macro like this-
addWebContent //parameters are title(required) and description(optional)
In java, the macro, if you don't want to set the description you can just do something like: addWebContent(title, null);
However Poshi won't check the absense or the presence of either, which test engineers can often make mistakes on. Missing a parameter will often lead to them having to manually debug the problem with their test (and with the common occurence of layering macros), a missing parameter may be very hard to find, and often wastes a ton of time. By making parameters explicit if we want to not set them, changing a macro's params, or even adding / removing certain ones will generate a compile error in Java (and visible errors within text editors such as Intellij), so test engineers will KNOW for sure that the parameters are being set correctly.
Poshi work around, check if every single variable required isSet:
if ((!(isSet(JSON))) || (!(isSet(key))) || (!(isSet(value)))) {
fail("Error, 'JSON', 'key', and 'value' must all be set");
}This statement for 3 variables can get very very very long if even more parameters are required. (think of how many potential parameters the Journal Service has).
public JournalArticle addArticle(
long userId, long groupId, long folderId, long classNameId,
long classPK, String articleId, boolean autoArticleId,
double version, Map<Locale, String> titleMap,
Map<Locale, String> descriptionMap,
Map<Locale, String> friendlyURLMap, String content,
String ddmStructureKey, String ddmTemplateKey, String layoutUuid,
int displayDateMonth, int displayDateDay, int displayDateYear,
int displayDateHour, int displayDateMinute, int expirationDateMonth,
int expirationDateDay, int expirationDateYear,
int expirationDateHour, int expirationDateMinute,
boolean neverExpire, int reviewDateMonth, int reviewDateDay,
int reviewDateYear, int reviewDateHour, int reviewDateMinute,
boolean neverReview, boolean indexable, boolean smallImage,
String smallImageURL, File smallImageFile,
Map<String, byte[]> images, String articleURL,
ServiceContext serviceContext)
throws PortalException {Imagine having to check every single one in Poshi. And since Poshi doesn't really support multi-line like java, the IsSet check will probably be around 200 characters long on a single line (breaks JavaLongLinesCheck) by a huge amount, but is forced like this with Poshi
Not having the flexibility to easily check for Parameters in Poshi that is built into many programming languages costs a lot of time for test engineers to make test fixes for un-set variables as well as debug issues that may occur out of refactoring the parameters of a macro.
Limited Operators Maintainability
There are cases when it makes sense to use common logic operators or control flow options such as iterators, XOR, or NOT EQUALS in functions. However many of these are not available on Poshi, so users will need to create them manually, which creates a lot of mess within the code.
Poshi XOR workaround:
@summary = "checks if either A or B is set, but not both"
macro _exclusiveOrCheck {
if (((isSet(A)) && (isSet(B))) || ((!(isSet(B))) && (!(isSet(A))))) {
fail("${failMessage}");
}
}Poshi != workaround:
if (!(("${addGuestPermissions}" == "true") || ("${addGuestPermissions}" == "false"))) {
fail("Invalid addGuestPermissions: '${addGuestPermissions}' specified. Allowed values are 'true' or 'false'.");
}Poshi For loop workaround (this way is much messier than a real for loop)
while (!("${ratingPosition}" == "${rating}")) {
AssertElementPresent(
key_ratingStarPosition = "${ratingPosition}",
locator1 = "Ratings#YOUR_RATING_STAR");
var ratingPosition = MathUtil.sum("${ratingPosition}", "1");
}Workaround to escape - sign
var curl = '''
${portalURL}/api/jsonws/fragment.fragmentcollection/get-fragment-collections
-u test@liferay.com:test
-d groupId=${groupId}
-d name=${collectionName}
-d <CURL_DATA[start=-1]CURL_DATA>
-d <CURL_DATA[end=-1]CURL_DATA>
-d -orderByComparator=
''';
Local API Calls Developers will likely be willing to help with this
Not all API calls are available remotely. This makes sense for production environments, with one of the best examples being activating staging. Staging cannot be activated remotely, likely due to the many steps it requires to actually activate it, which would be very difficult to do with 1 single API call (and possibly for security reasons as well). However, none of those actually matter to us in a testing environment. It would be a huge help to our tests to activate staging through an API. However Poshi can only make Remote API calls. And even if it were possible to make a call to activate staging to activate the API, creating the data necessary to do so through Poshi would be a huge nightmare and would require a large amount of hardcoding.
Java testing would allow us to simply import the required modules to activate staging. These macros wihin a TestUtil could be used in both integration testing and functional testing.
Using a Debugger
The power of a debugger such as Intellij is huge. Trying to debug Poshi is in fact a nightmare.
- Break points don't exist
- Poshi cannot pause on the fly.
- No access to local variables due to no debugger
- Debugging Poshi often results to watching Poshi for the entire test run, which can often be 15~minutes, and a single glance away could result in all that time being wasted because the tester missed the test failure.
A lot of time is spent debugging Poshi tests, and with no real debugger other than using a bunch of print statements, this often leads to a lot of wasted time.
There probably is not simple workaround as we would likely need to create a working Poshi plugin within Intellij or some other editor and also maintain it, and would likely take a long time to actually work.
Syntax Checking
Poshi does not offer much, if any help at all with syntax checking. Forgetting a semi-colon, a comma, or a bracket can be easily be caught within an IDE with a real programming language, but Poshi usually only tells the correct file the syntax error occurs on, with very low accuracy, or no help on where the error is located at or caused by. This can often lead to a single syntax error causing 1-2 hours of time loss when it could normally be found within seconds with the help of a Java IDE such as Intellij.
Public/Private class modifiers
In Poshi, sometimes paths or macros are created very specifically for another macro, or set of macros for a specific "object" For example, maybe deleting a WC, because its implementation is different than others. In Java, we could just set methods / paths (variables) to private if we don't want others to access it, but in Poshi, everyone can access everything, and no imports are required. Imagine how messy integration tests would be if every thing was made public and nothing needs to be imported. Thats basically what Poshi is like right now. Modifying a macro you made for a specific class could easily break something else that is using it, but was not even supposed to use it in the first place. Others have no idea if they should use a specific method or path or create their own because there is no way for them to know if it is expected of them to use it or not. So without these rules, our testing macros and paths become like a spider web / anarchy with no rules and constant chaos everywhere with every fix. One fix may require a tester to check 10-20 other test to see if we broke their changes, all because we don't have a way to explicitly declare if our macros/paths should be useable from outside classes.
Class Extending (Perhaps just an interface is better for base class) Object composition was mentioned as well
It may make sense that every add button in liferay has the same locator, every management toolbar is the same, every modal has the same iframe locator. Well they do not unfortunately, even for many taglibs. Because of so many different front-end issues can occur, there are a lot of patches or variations within seperate buttons and classes, so buttons for WC and Blogs may differ greatly, however certain parts may also be the same for them. So what ends up, is that there are many, many macros for every single UI component with no real definition to them.
Take the management toolbar for example. It is known that there is a base management toolbar that should be the same for all components. Search field, Add button, and icons to use Order By, Filter by etc. However, there are also a lot of differences across components depending on their implementations.
If Object Oriented Programming was allowed, we can simply have a BaseManagementToolbar macro class, that can convey to every tester of what can vary, and what would be generally the same. Here is an example.
public abstract class BaseManagementToolbarMacro {
public static void clickPlus() {
Click.clickPlus();
}
... //etc
public static void filterBy(String action);
public static void orderBy(String action);
private String[] _actions;
private String[] _filterByOptions;
private String[] _orderByOptions;
}This is super powerful and reduces a lot of maintence and confusion from test engineers.
- Having the set options in the bottom can be used to make sure that the available options for each module is known without any doubt, and can be used to verify that incorrect actions are not performed.
- Any base commands that are usually the same for every component such as clicking the plus button could be easily changed if the base front end change is applied, fixing the issue for multiple components at the same time, or allowing them to override it if they dont need the change (Normally this would require a lot of searching, with still a bunch of uncertainty due to no way to be sure what macros are related to other component's management toolbar because they are not attached to any class)
- There are often times when a front-end component may get upgraded for modules one at a time, for example 2 years ago when the management toolbar update was applied 1 component at a time, starting with WEM. Previously a whole new set of macros needed to be created, and then components that the changes got applied to would have to do a mass search + replace to apply those changes, and made sure that the changes were correct, and then repeat. And we would also need to remember to remove the old macros once the last management toolbar update was finished. (maybe they are still present in Poshi now). If we had used a base class like this, the management toolbar update would have went through seamlessly with few test fixes required.
- New components can easily use these base classes without having to worry too much about missing macros
- Macros are organized a lot better now (currently there is no set organization) and can easily be found
File Object
The file object is required to be able to invoke certain API calls that require a file, often for Documents and Media, Blogs, and sometimes WC. Poshi does not support the object, and there is no way to retrieve the byte data from the file (if theres a method that just requires the byte data), so it is impossible currently to invoke API calls in Poshi that require a file.
Access to Liferay Globals
There are many occurences that certain Liferay Globals need to be accessed in functional tests for API calls, navigation, or other various occurences. Because Poshi has no ability to import any of the Liferay modules, accessing globals is not possible and they need to be hardcoded. However this is very bad practice for 2 reasons. 1) if the globals ever change, the test/s will now fail 2) There is no way for users to know what the hardcoded string is supposed to reference.
Here is an example where we should have just retrieved the Portlet Ids, but instead we have to hardcode them:
if ("${widgetName}" == "Asset Publisher") {
var portletId = "com_liferay_asset_publisher_web_portlet_AssetPublisherPortlet";
} else if ("${widgetName}" == "Search Results") {
var portletId = "com_liferay_portal_search_web_search_results_portlet_SearchResultsPortlet";
} else if ("${widgetName}" == "Type Facet") {
var portletId = "com_liferay_portal_search_web_type_facet_portlet_TypeFacetPortlet";
} else if ("${widgetName}" == "Web Content Display") {
var portletId = "com_liferay_journal_content_web_portlet_JournalContentPortlet";
} else {
fail("'widgetName' is either invalid or its 'portletId' is not yet set.");
}With front-end developer help, decide what scripts need to be loaded before interacting with certain objects instead of waiting an arbitrary amount of time
Waiting a random duration before a component cause a lot of un-necessary waiting and is unstable if we are not super safe because performance can easily fluctuate. With access to easy modify the base java level directly, we can apply this to many front-end components known to fail using a class model while avoiding having hundreds of poshi "functions" like these
function clickAtSidebarClickAtWaitForScript {
WaitForSPARefresh();
selenium.pause("1000");
selenium.waitForElementPresent(
"//*[@data-qa-id='controlMenu']//*[@data-qa-id='add']"
);
selenium.mouseOver();
selenium.waitForElementPresent(
"//script[contains(@src,'/liferay/node.js')] | //script[contains(@src,'/js/control_menu.js')]"
);
selenium.pause("1000");
Click.clickAt();
selenium.waitForElementPresent(
"//script[contains(@src,'/liferay/dockbar_add_application.js')] | //script[contains( @src,'/js/product_navigation_control_menu_add_content.js')] | //script[contains(@src,'js/product_navigation_control_menu_add_application.js')]"
);
}
Edits to BaseWebDriverImpl (adjusting / creating new required methods) can take a long time days-weeks https://issues.liferay.com/browse/LRCI-160 https://issues.liferay.com/browse/LRQA-48433
Release of Poshi versions take a while, Request/Implement on Own, Wait for Pulls to Foward, Wait for Poshi Release
This process will likely not change as long as we are using Poshi