There are more than twenty test automation tools and frameworks. Let’s explore a tried and true method for identifying the right tools for your next test automation project.
By: Julia Pottinger
There are more than twenty test automation tools and frameworks, including Selenium, WebdriverIO, Cypress, Nightwatch, Playwright, and Robot Framework. The list may seem endless and overwhelming, but I’ll share a step-by-step process to help you evaluate these options and find the right fit for your test automation project.
Let’s begin with a simple scenario:
Scenario
Your team has developed an application that allows users to create bugs for popular apps and receive bug bounties based on the severity and reproducibility of the bugs. This application has a web version as well as a mobile Android and iOS native application. The web application is compatible with all browsers, including Internet Explorer, Safari, Chrome, Opera, Firefox, and Microsoft Edge and has iframes on each page. Links open in new tabs. The application uses APIs to collect and store large amounts of data in a database. The application interface varies slightly between apps; for example, Twitter uses different colors and text prompts when creating bugs than Instagram.
Imagine your team wants to add test automation to improve the release process and reduce the number of errors in the application while increasing test coverage, catching new and existing bugs, and integrating tests into a CI/CD pipeline. Your team also wants to increase the number of browsers and devices that can be tested simultaneously without drastically increasing the number of testers or real devices needed to execute the tests.
Here’s how you decide what tools and frameworks you’ll use to automate the project.
Step 1: Make a list of features and automation requirements
Create a list of features the tools and framework will need to support.
For example, using the scenario provided, write down each feature and its corresponding requirements:
- Cross-browser testing: compatible with all browsers
- Mobile automation on both iOS and Android devices: mobile Android and iOS native application
- Tab and iframe support: iframes are on each page and links open in new tabs
- API Automation: uses APIs to collect the data
- Data-driven testing: collects and stores large amounts of data, and the interface for the application slightly differs per popular app
- Database automation: collects and stores large amounts of data
- Visual Regression Testing: uses different colors and text prompts when creating bugs
- Unit Testing for features: creates bugs for popular apps and receives bug bounties
Now that you have a list of requirements, it’s time to evaluate the scope and size of your project.
Step 2: Evaluate the scope and size of your project
Decide which project requirements are in scope for your automation. Create a test automation strategy to identify the goal and scope of your project and to clarify outcomes. Perhaps you’re automating six features of the web app supporting Chrome and Firefox browsers or only doing API automation, identify your goal early.
The test automation project presented in the scenario above has many moving parts, including web and mobile automation on several browsers and mobile devices, API automation, and Database automation mixed in with data-driven testing. The features of the application include authentication, bug submission, dashboard with bug bounty status, payout of bug bounty, security, and performance. The project scope is complex, so simplify by focusing on smaller parts, and increment the automation project over time to discover what tools are essential.
Using your test automation strategy:
- Scope the project and identify the features that require automation.
- Create a list of must-have automation scenarios.
- Determine the implementation timeline.
- Identify the key players involved with project implementation and maintenance.
Now that you’ve determined the scope and size for your project, it’s time to identify the application platform you’ll use to build your project. Use this insight to create a shortlist of tools and frameworks.
Step 3: Identify your application platform
The platform you choose to automate will influence the tools you use. You’ll use a different set of tools to automate mobile applications, like Appium, TestComplete, Espresso, XCUITest, and Robotium, than you’ll use to automate UI tests on desktop browsers. There are also different tools for automating APIs, databases, or web applications.
Step 4: Consider programming languages and frameworks
Your automation engineers must understand the different tools and frameworks they’ll use to write the script. If they are well versed in a particular language and framework, then add those options to the shortlist.
When selecting automation engineers for a test automation project, evaluate their knowledge and expertise in test automation best practices. They should understand:
- Data management and database automation;
- How to select the correct tests to automate;
- How to identify the required level of test automation; and
- How to use API and database automation to set up and populate scenarios for UI automation.
If the team is well versed in Gherkin style and wants to allow manual testers and product owners to use Gherkin to write out automated test cases, you must account for this in both the framework and tools you select. If the programming language of preference is Javascript, then the tools should lean toward that language. Additionally, based on the expertise of the team, you may need to consider low code tools that use automated scripts and apps versus manual code.
Step 5: Calculate costs
Set a budget for your test automation project. Account for the size of your staff, how much you’ll spend on salaries and for how long, how much you’ll spend on tools and infrastructure, and project maintenance costs.
Then, decide whether you will pay for a test automation framework or use free, open-source tools supported by the community. There are pros and cons to each option:
- Open-source tools: while free to use, are dependent on community support and maintenance. These tools are constantly evolving, and you have an opportunity to contribute to their improvement over time. However, without continued community support, open-source tools become outdated.
- Commercial tools: offer a lot of features out of the box, but they can be very pricey. Community support is not as easily accessible and tutorials and training may be included at an additional cost.
Step 6: Research and compare existing tools
Find out what tools are popularly used by automation engineers and other teams within the company. What are the developers using for unit testing? Is there an existing continuous integration pipeline where automated tests can be recorded?
Tools and frameworks can be separated by test level, so there may be different tools based on whether you are doing API, UI, Database, etc.
Identify which tools are currently being used by your team for each of those areas and evaluate the current effectiveness of these tools.
- Are there any pain points?
- What is going well and what can be improved?
Use these findings to determine if those tools are good candidates for use in your test automation efforts.
Step 7: Create your shortlist
Using the scenario provided, your shortlist of tools should begin to look like this:
| Tool/Framework | Javascript Programming Language | iFrame Compatibility | Mobile Automation | Cost | Browser Support | Community and Learning Support |
|---|---|---|---|---|---|---|
| Selenium WebDriver</ a> |
✅
|
✅
|
✅
|
✅
Free |
✅
|
✅
|
| WebdriverIO |
✅
|
✅
|
✅
works well with Appium for mobile automation |
✅
Open source and free |
✅
|
✅
Lots of community support |
| Cypress.io |
✅
|
⚠️
|
❌
|
⚠️
Free except for the Cypress Dashboard |
⚠️
Chrome Family Browsers(Edge and Firefox) |
✅
Lots of community support |
| Playwright |
✅
|
✅
|
⚠️
Uses device emulation to test responsive web apps in mobile web browsers |
✅
|
⚠️
Playwright does not support legacy Microsoft Edge or IE11 |
❌
Pretty new; not as much community support |
| Codecept |
✅
|
✅
|
✅
works with Appium for mobile automation |
✅
|
✅
|
⚠️
not as much as other frameworks |
Add other frameworks and comparison points as needed.
Consider whether your team is familiar with the tools and frameworks. By using familiar tools, you can achieve quicker setup and faster automation.
Using this table I would select Selenium Webdriver, WebdriverIO and Codecept to move forward with to the next step. These frameworks meet the requirements listed for the project.
Step 8: Create a Proof of Concept (POC)
Once you have a shortlist of frameworks, select your top three and create a Proof of Concept (POC). This POC can help you identify:
- How the framework is maintained;
- How easy it is to use;
- Whether it addresses your requirements; and
- If it addresses some of the high-priority edge cases you have, as well as different scenarios and controls in the application.
The POC helps you identify risks that may exist with the test framework and the tools you have selected. Ensure that you have a checklist of items to answer during this POC process, then decide if the tool is worth using going forward.
Key takeaways
There are many factors to consider when selecting tools and frameworks for test automation. You’ll need to be patient with this process; however, you can save time, money, and headaches in the long run by taking this strategic approach.
Remember that each tool is different; think about what you want it to solve, who will be using it and what their competencies are, and how much the tool and maintenance of your project will cost. Most importantly, do a Proof of Concept to ensure the tool is sustainable and meets the edge and important cases you are looking for.
QualityWorks provides test automation services, as well as assessments that help you evaluate the current needs of your company. We help you identify opportunities for optimization by conducting a transformation maturity assessment of your team, processes, tools, and culture. Contact us today to learn more.
About the Author
Julia Pottinger is the Head of Training and Development at QualityWorks with expertise in manual, automated, and API testing and training. She is passionate about sharing her knowledge and experience with new developers and contributes to the testing community by writing articles, delivering testing content on Test Automation University, and producing her Youtube Channel and blog. She also conducts testing boot camps for people interested in entering the field of Quality Assurance.