Automated Testing

Software teams are constantly paying attention to ways that they can become more efficient. They set goals to ship code faster, better, and with less overhead. Some teams can add more members to help combat a growing customer base and product. Whether a team is expanding or not, automating tasks can help a team free up more time to expand by getting more out of the limited time they currently have.

Thankfully, for team leads and engineers, many software tools are available on the market to help free up their developers. However, automation is no longer just the responsibility of developers. The advancements in user-interface design and testing technology are making automation tools more accessible for team members with less technical skills.

Developers and quality assurance (QA) work together to deliver new products. The back and forth between developers and QA can be a time drain on an organization. An answer to improving the efficiency of the entire process is to provide tools that help automate testing for technical and non-technical users.

Furthermore, some teams may not have the luxury of dedicated QA. It’s even more important for these teams to automate testing and integrate CI/CD pipelines with automatic testing tools.

Related: How to Automate API Tests with RapidAPI

Types of Testing

Different applications have different testing needs. In this section, we are going to take a look at some of the most common types of testing that automated software testing tools target.

Functional. Checks whether the software is satisfying requirements. Functional testing makes sure the software module is fulfilling its purpose within the application or process.

Regression. According to Software Quality Assurance, Testing and Metrics, regression testing is done to ensure that existing features are not broken by new features. This is achieved by re-running functional tests.

Unit. Test specific module, code, function using a defined operating procedure, data, and execution environment. Learn more about unit testing.

Key-word. Keyword-based testing is a framework for building tests constructed around actions that are taken by a user of the software. Keyword-based testing could follow a user-object (representing the user interacting with the application) as they “login” and “access” data with the application.

End-to-end. A clear and succinct definition of end-to-end testing can be found on Tutorialspoint: “End-to-end testing is a technique used to test whether the flow of an application right from start to finish is behaving as expected.”

If you are testing the user-interface of an application it may make more sense to target end-to-end or key-word based testing. On the other hand, testing raw data favors a functional or unit-based testing approach.

Check out a few API Testing tips here.

How to Choose a Testing Tool

Now that we have discussed different types of testing, let’s talk about choosing the right testing tool. Many organizations know what they are looking for in a testing tool because they understand their problem.

When vetting testing tools there are features and pricing tiers to pay attention to. This will help ensure that you are selecting the proper tool.

What application are you testing?

First, as previously mentioned,  tools are often providing certain types of testing with their software. They’ll often state, on their website, whether the tool uses keyword-based testing, focuses on end-to-end testing, etc. Additionally, if you are testing an API, some tools in this list wouldn’t make sense because they are based on testing user-interfaces.

If you have both a user-interface to test and an API, then you should consider if one tool handles both of the testing needs or if you can combine two tools.

Scalability vs. Price

Before choosing a testing tool you need to consider:

• How many users are going to need access to the software?
• Where is the software going to run?
• How many tests will you need to execute (parallel testing)?
• What are the essential features for your testing case, and what would be the ideal features?

These questions are core considerations when evaluating how much you will actually end up paying for the testing tool. The pricing tiers for some automation tools are based solely on the scalability of the testing process.

Usability

Testing tools exist on a spectrum of how much technical knowledge it takes to operate the application. Will the skill level of the members using the software be sufficient to understand this tool?

Additionally, the user interface (UI) of the application plays a role in how easily a tool will be accepted into your development process. A clean UI says a lot about what kind of a product a company is trying to provide.

Finally, double-check that the tool has sufficient documentation to assist developers or QA with core features of the software. Waiting to hear from customer service can be exhausting. If users can self diagnose their issues, and understand their use case, they waste less time trying to figure things out on their own.

Best Automated Testing Tools

1. RapidAPI Testing

Create and manage comprehensive API tests from development through deployment

RapidAPI Testing is a new solution offered by RapidAPI for monitoring, integrating, testing, and automating APIs. With RapidAPI you can take your team’s APIs, which may already be on RapidAPI’s global API market, and have the application parse the API to quickly build test cases.

Technical or non-technical users can create complex tests using the Request Generator, to execute assertions on different API routes, or run custom Javascript scripts. The service is accessible in the browser and tests can be run from nine locations throughout the globe.

RapidAPI Testing exposes a dedicated webhook for CI/CD integration with Jenkins, CircleCI, GitHub, Travis CI, and GitLab.

More features from RapidAPI:

• Schedule API tests from different locations and environments
• Set SMS or Email alerts for failed tests
• Monitor API performance based on response size and request latency
• Supports REST, SOAP, and GraphQL

Plans and Pricing

You can execute up to 100,000 API calls free with RapidAPI Testing. All plans offer an unlimited number of test cases. The ideal subscription is the Ultra package that has:

• Email and SMS notifications
• 1,600,000 API calls
• CI/CD webhook integration

This plan includes all the features mentioned above with no additional licenses or subscriptions. The plan starts at $159/user/month. You can send more API calls with the Mega subscription, and there is a custom tier that allows for an organization to install the RapidAPI Tester.

Although the Ultra plan is the most popular, the Pro plan still offers email notifications, CI/CD integration, and three times the API calls compared to the Basic subscription.

2. Selenium

The Selenium Project is open source (view Governance document) and many tools have been built using the Selenium software. There’s no cloud subscriptions or plan tiers.

Selenium has a perfect introduction on their website.

Selenium automates browsers. That’s it!

What you do with that power is entirely up to you.

My first introduction to Selenium was writing Python code. I used one of Selenium’s web drivers to open a browser and interact with a website. The Selenium WebDriver,  “…drives a browser natively, as a real user would, either locally or on remote machines.” Next, let’s review the tools Selenium offers to shed light on how much technical knowledge is needed.

There are three different tools that Selenium provides:

1. WebDriver: the web driver is “non-intrusive”. This means that it will control the browser, like a user, and interact with your web application. To use the web driver, you’ll need to install the bindings for your programming language. Then, install the browser binaries.
2. IDE: The IDE is a browser extension that records actions made by a user in the browser. Then, Selenium tries to develop test cases based on the interaction. This is a “time-saver”, and it beneficial for newcomers to Selenium.
3. Grid:  Grid is the next step for someone that has developed their own test cases using a programming language and web driver. It allows test cases to run across different platforms and browsers.

Selenium is different from many of the other tools in this list. It’s open-source and requires technical programming knowledge. However, the need for technical knowledge is a symptom of its flexibility and customization capabilities. Selenium software is always a consideration for automated testing.

3. Katalon Studio

It makes sense to follow up Selenium with Katalon Studio. That’s because Katalon Studio is an organization that builds testing tools with open-source software (like Selenium and Appium). Katalon Studio provides tools for web, mobile, and API testing.

To use Katalon Studio, you need to download the program onto your local machine. Katalon offers an IDE or Advanced IDE for Enterprise licenses. IDE stands for Integrated Development Environment. The IDE provides a graphical user interface (GUI) so developers or less technical QA users can build test cases, test suites, and testing projects. Katalon Studio tries to provide a GUI that helps users better leverage open-source tools.

Plans and Pricing

You can download a limited version of Katalon Studio for free. This provides testing across web, API, mobile, and desktop. To get advanced features you’ll need an enterprise license. However, inside the Enterprise license, there is a node-locked license (one user, one machine) and a floating license (license shared across multiple machines). The price doubles, from $759 to $1,529 per year between the node-locked and the floating licenses.

It’s also important to note that support for all execution environments, enabling CI/CD, and test scheduling is purchased with an additional license. The Katalon Runtime Engine license also is split between floating and node-locked with the price starting at $539/license/year with node-locked and going up to $1,199/license/year with floating.

This means, to the best of my knowledge, that to have the ability to schedule tests using Katalon you’ll have to buy the Runtime Engine license with either the free or enterprise version of the IDE. The pricing details are a little foggy with variations in license type and users, but it appears that multiple licenses will be required.

4. TestComplete (SmartBear)

TestComplete is a SmartBear product for creating and running UI tests.

The service offers:

• Test playback
• Scripting
• Script-less test building capabilities
• Integrations with common application software (i.e Git, Jenkins, Jira)

TestComplete emphasizes object-recognition in their testing. The software tries to identify objects and attributes that cause tests to fail. I’ve seen something similar when using Visual Studio in a .Net project.

TestComplete tries to thread the needle, as many services do, between a high-tech and low-tech workplace. Based on images, the user-interface doesn’t look like anything to get excited about. However, there appear to be many features packed in that integrate with other products and services that SmartBear offers.

Plans and Pricing

SmartBear is a company that has many products, licenses, services, etc. It’s difficult, at times, to figure out exactly how many licenses and products you will need.

You can buy TestComplete Base ranging between $2,200-$9,700. This will vary depending on whether you want node-locked or floating and on the devices that you want to test on (desktop, mobile, web).

However, to execute tests created in TestComplete on devices that don’t have TestComplete installed you will need to buy a license for TestExecute ($499/year/license). If you want to cross-browser test then you’ll, also, need CrossBrowserTesting by SmartBear.

TestComplete is for testing user-interfaces. You can expect to double your costs if you want to add licenses for testing your API with SmartBear products like ReadyAPI or SoapUI.

There are many different ways to bundle licenses discussed that could affect the price. The full breakdown of pricing is beyond the scope of this article.

5. LambdaTest

LambdaTest is a cross-browser testing and compatibility tool. The service is cloud-based and accessed in the browser.

From the LambdaTest dashboard you can:

• View mobile and desktop web applications from different operating systems and devices
• Run tests across multiple browsers
• Integrate with different applications for bug tracking, communication, CI/CD, and other test automation tools like Katalon Studio and Ranorex.
• View test logs

The user interface is clean, making it easy to navigate and find features.

LambdaTest separates the live testing that is done in the dashboard from automation testing. Automation testing is set up through Selenium’s Grid tool (that we talked about earlier in the article). This means that you can run Selenium scripts (developed in your language of choice) on LambdaTest’s cloud-based Selenium testing grid online.

This may increase the setup time for getting your tests automated with LambdaTest, but I think it adds flexibility to the service.

Plans and Pricing

The ideal LambdaTest subscription is their Web & Mobile Browser Automation. The base plan covers 1 parallel test for five users and gives you unlimited web/mobile browser automation testing, real-time browser testing, and third-party integrations.

The plan costs $99/month and increases the more parallel tests or users you add.

6. TestProject

Free Test Automation for All.

testproject.io

TestProject is a test automation platform for web, mobile, and API testing. To use TestProject, you need to download the TestProject agent for your given OS. The agent, “…is a powerful wrapper for Selenium and Appium and with one simple install you instantly have access to all browsers and connected devices on the machine you install it on.” 

TestProject combines local testing given a device/emulator (through the agent) with a cloud component for creating and sharing tests. The project provides codeless solutions to record, store, execute, and analyze tests on desktop or mobile.

It exposes a REST API for integrations with other software. This may require more work than paid options, but it still leaves the door open for integrations.

The platform has a community that has created add-ons of tests for different file types, actions, platforms, devices, etc.

7. TestArchitect

TestArchitect is a codeless testing automation solution that can be extended to support more programming languages. The product is delivered with an IDE that looks similar to the TestComplete IDE.

With TestArchitect’s IDE, a user can:

• Create codeless test cases
• View test analytics
• Integrate with Jira, Jenkins, and other common CI/CD software
• Record UI Actions
• Run parallel codeless tests

Action Based Testing (ABT) is a keyword-based testing framework designed by TestArchitect’s parent company LogiGear. The company claims that ABT, “…makes it possible to automate 95% of tests while using only 5% of the total test effort for automation.” The approach can help users organize their test cases based on customer actions (“stories”).

TestArchitect is another service that can be paired with Selenium for cross-browser testing.

Plans and Pricing

TestArchitect offers two plans. They have a free plan that allows for IDE use on two machines (node-locked) and up to one-hundred test cases.

Their Team & Enterprise plan doesn’t provide any pricing information but can be purchased with floating, node-locked, or run-only licensing.

8. APIFortress

APIFortress is an organization that offers solutions for API testing, monitoring, mocking, and automation. The last few tools discussed so far have focused on automated UI testing. This platform focuses on testing APIs and microservices.

With APIFortress, the user would upload an API specification. Then, they could build functional tests based on the routes, parameters, etc. of the API or microservice.

APIFortress provides capabilities for:

• Viewing test analytics
• Building alert groups and setting up notifications
• Automated test generation (from API spec)
• Integration with CI pipeline (Jenkins, webhooks)

Plans and Pricing

APIFortress offers a hosted cloud plan or a self-hosted cloud plan. The benefits of choosing the self-hosted cloud plan are:

• mocking
• load-testing
• Single sign-on

APIFortress does not publish their plan prices. You must contact them for a quote.

Browse API Fortress Alternatives

9. Postman

Postman is an API platform that offers an API client, monitoring, mocking, testing, and documenting. The software is accessible with the Postman desktop application available on major operating systems.

Related Resources:

• Insomnia vs Postman vs Paw
• Convert Postman Collection to OpenAPI

With Postman, you can import an API, create tests, build test suites, and then integrate those test suites with your CI/CD pipeline.

An important difference between Postman and other API testing/automation tools is the process for creating tests. Postman provides an API for writing test scripts to assert on the returned values for endpoints.

However, the Postman API uses Chai.js, a Javascript library for scripting tests. This would favor users with development backgrounds in Javascript, but would not be friendly for quality assurance agents or developers coming from a different programming background.

Plans and Pricing

Postman displays its pricing and plans in an easy to read and understand way. Postman has a Team and Business plan.

The Team plan costs $15/user/month. The Business plan is double the Team subscription but offers single sign-on and access controls.

10. Ranorex

Ranorex is a functional UI test automation application. The company offers products for desktop, mobile, web, and cross-platform testing. The tool offers object-recognition in UI testing with Ranorex Spy, an action recorder, and a code editor for extending test cases.

Ranorex’s interface is capable of providing point-and-click test building and scripting. This is suitable for technical and non-technical users.

You can integrate Ranorex with a preexisting Selenium Grid or many other software management tools like Jira, Git, Azure DevOps, BrowserStack, Jenkins, etc.

Plans and Pricing

You can buy a perpetual license for Ranorex Studio starting at $3,590 for a node-locked license (one user) or $5,990 for a floating license. You can renew the perpetual license for $700-$1200 a year for maintenance and support services.

Ranorex offers additional tiers bundling support and maintenance for premium and enterprise licenses. However, it doesn’t appear that you are paying for more studio features with the upper tiers.

11. mabl

mabl was founded in 2016 and provides UI test automation tools for quality assurance and developers. The application claims to provide codeless test creation and maintenance for non-technical team members. Although the company offers the ability to add advanced logic to your tests with Javascript snippets (assertions, variables, loops, etc.), it appears they believe the solution is to use less code.

mabl has a Chrome browser extension for clicking through the UI of an application to record and generate tests.

You can also expect popular integrations with technologies like GitHub, Bitbucket, Jira, Azure Pipelines, and Slack.

A couple of other features worthy of mention include:

• command-line interface for writing and executing tests locally available on npm
• automatic setup, maintenance, and teardown for test infrastructure
• technology stack agnostic

Plans and Pricing

mabl offers three plans: StartUp, Growth, and Enterprise. Test automation is available for all three plans, as well as integrations with Slack, Jenkins, and Slack. Interestingly, Selenium script import is only available on the Enterprise plan.

End-to-end testing is exclusive to Enterprise plans. This includes API testing, e-mail testing, and PDF testing.

The company does not display pricing information. Pricing plan costs can be requested on the company website.

Conclusion

There are many testing tools available to automate software tests. Be sure to not only do your research when looking for a tool but also do the research to understand what your organization really needs. Thanks for reading!

FAQs