Comparison of Automated Test Tools

“Progress isn’t made by early risers. It’s made by lazy men trying to find easier ways to do something.” –

Robert A Heinlein

It is generally understood in software development that any sufficiently complex solution will have bugs – and that threshold of “sufficient complexity” is lower than most people assume. Flushing out these bugs is a central task in development: Quality Assurance’s initial raison d’etre. QA philosophy has since expanded and evolved, but its fundamental, most concrete concern persists.

Traditionally, bugs have been found by manual testing. If a tester wanted to know whether a page loaded with all its expected elements, the tester would enter the URL to compare the page returned with a test case or a requirement document. Any missing element had to be seen, identified, and reported via human interaction. This methodology is workable, but it has several inherent drawbacks:

• It depends on the concentration/accuracy of the tester
• It involves the labor cost of having the tester perform the check
• If there were intentional but undocumented changes to the page, it may result in the reporting of a bug where none existed.

Test automation is a controversial but potentially invaluable attempt to mitigate these concerns. An automated test cannot suffer from lapses in concentration (or any other psychological failing). A suite of automated tests can be run in a fraction of the time it would take a tester to run one manual test. Automated tests can even learn to recognize elements on a page based on the underlying data/object model, so that they won’t necessarily be confused/stymied by last-minute changes to a product’s design. There are a multitude of test automation tools designed to help dev teams secure these benefits when they automate their tests. In this blog, we’ll examine a few of the industry-leading software packages.

Selenium is unavoidable in the world of test automation; in fact, many of the other tools mentioned here include (or even recommend) integration to piggyback on its capabilities. It’s an open-source tool for automating testing of web applications that is able to run on Windows, Mac, and Linux operating systems and functions in Chrome, Firefox, Internet Explorer, and headless browsers. Selenium tests can be created via recording of human actions using the Selenium IDE Firefox extension. Tests can also be scripted using Java, Groovy, Python, C#, PHP, Ruby, and Perl. SeleniumGrid offers data-driven testing capabilities for re-running a test with multiple predefined datasets. Selenium undergoes constant maintenance and improvement. Most recently they’ve made sure their simulated testers can navigate uploading/downloading data, popups, and dialogs.

Katalon Studio is a free test automation tool for web, mobile (Android and iOS), and web services/APIs. It is able to test in Chrome, Firefox, Safari, IE, Edge, and headless browsers on Windows, macOS, and Linux systems. It is designed to integrate with Git, Jira, qTest, Kobiton, Slack, Sauce Labs, BrowserStack, and SeleniumGrid for everything from chat to data-driven testing. It provides for the creation of run-time rules to handle test failure and complex use flows, preventing the suite from getting stuck when it first encounters bugs. Katalon also recently released a beta of their “Katalon Analytics” package, allowing for dashboard metrics and reporting on test results and the bugs found.

Unified Functional Testing is a commercial test tool from Hewlett Packard’s MicroFocus division designed for cross-platform testing of the APIs, web services, and GUIs of desktop, web, and mobile applications. The tool is designed to integrate with HP’s Business Process Testing (formerly Mercury Business Process Testing) and HP Quality Center (formerly Mercury Quality Center), as well as various third-party tools, such as Jenkins’ CI (Continuous Integration) functionality. Tests can be defined via manual recording or written using VBScript. UFT uses keyword-driven and element-driven testing to maintain highly distributable automated tests and can run its test across many browsers and mobile devices without the need to rewrite them.

Watir provides an open-source Ruby library for web-based test automation capable of running its tests in Firefox, Opera, IE, and headless browsers. As of January 24, 2018, there is also a Python equivalent of the Watir features dubbed Nerodia. While quite robust, the Watir services require more work to set up: IE testing requires integration with Selenium, Chrome testing requires ChromeDriver, and Firefox testing requires geckodriver. Generally speaking, Watir encourages basing its implementations in Selenium for compliance with the W3C test automation specification.

IBM’s Rational Function Tester is a commercial automation tool designed for data- or GUI-driven functional and regression testing. It is capable of testing web-based, .Net, Java, Siebel, SAP, terminal emulator-based, PowerBuilder, Ajax, Adobe Flex, Dojo Toolkit, GEF, Adobe PDF document, zSeries, iSeries and pSeries applications. Scripted tests are written in VB.NET and Java, but manual recording of test steps is available as well. RFT includes a test “storyboarding” feature to determine the flow of tests and capture images of each stage, providing visually intuitive test-planning. RFT’s ScriptAssure uses fuzzy matching of object properties with adjustable weights to locate the same objects across different versions and contexts, increasing tests’ resiliency in the face of design/requirement changes.

If you’d like to utilize TechArk’s high-quality, cutting-edge development and QA processes to create your own website or application, get in touch with us today!